Combining automation with ML

How to do it humanely & effectively.

Initially, treat the ML as an advisor that makes suggestions the human experts can take or ignore based on their own judgement. They’ll accept it most of the time but will need to improvise in the long-tail of unusual situations. The manual overrides should be used as signals to continue training the model.

As the model improves to the point where it’s better than all but the best humans, have it act on its own suggestions by default when its confidence level is very high. This will allow the humans to focus on the more interesting situations. But the automation should always leave a detailed log of why it makes each decision in case something goes wrong & needs to be investigated.

Now the original pool of human experts can be slowly enlarged by using the model to train new humans! Have the new human try to guess what the model did in a weighted sample of past decisions until they’re roughly as accurate as the model, albeit much more deliberate. Now have them shadow the existing human experts as they tackle the unique corner cases until they can consistently predict even those decisions.