Eric Siegel has a book devoted to getting the best out of AI; specifically its prediction-related branch, machine learning. He gives lots of practical advice on machine learning development. How then to get the best out of your algorithms?
Practical Advice On Machine Learning Deployment
Siegel gives a way of thinking about machine learning development. This is his process called, BizML Practice. In essence, the books unveils a framework for thinking about how to get the best from your algorithms. I must confess to never having implemented a machine learning system, but his advice seems sensible. He emphasizes the need to link the model to the strategy. Implementing a model should be seen as a business problem, not a modeling issue. Start your objectives by saying what “we want to do”, rather than what you want “the model to do”.
In general, it is really important to know why you want to do something before doing it. That seems very sensible.
He also notes where to expect the problems. Wrangling the messy data is a perennial source of strife.
Data preparation is a perpetually underestimated bottleneck…
Siegel, 2024, page 205
This seems very plausible given the related tasks I’ve completed. You might often develop the impression that getting the data ready is an easy task — perhaps you could ask an intern could do it. Yet, working with messy data can be a nightmare. Plus, if you do it wrong, any model that is developed from the data won’t be worth much.
Leaks From The Future
One data challenge Siegel highlights is that of ‘leaks from the future’. Such leaks give the model (algorithm) information that helps it work out the answer. Yet, this information shouldn’t be available as going forward it won’t be there when the model is implemented. The example he gives is of a model being created on historic data that attempts to predict churn, i.e., loss of customers. Even if you don’t tell the model which customers were retained if the customer is on a later customer mailing list the model will work out that this customer was retained, at least for a little while. Unfortunately, the model is using information it will not have when it is implemented. This means the model will likely be really good at predicting what will happen in the current data set, but likely won’t do half as well when it does not have the cheat code.
Model Drift
A final concept worth remembering is that of model drift.
Over time, your model inevitably devolves into a defunct dinosaur, a phenoneon known as model drift.
Siegel, 2024, page 207
Sadly completing a perfect implementation isn’t enough. You need to go back and do it again when the model becomes out of date. You need to keep an eye on your model. I guess artificial intelligence is becoming more and more like us. There is a learning phase, a painful transition to being useful, and just when all that work starts to pay off you find that you are obsolete.
Read: Eric Siegel (2024) The AI Playbook: Mastering the Rare Art of Machine Learning Deployment, MIT Press
