Steven Finlay has a useful book on Data Science, (Predictive Analytics, Data Mining and Big Data). He has lots of helpful practical advice in an easy to access form. He highlights the idea of the flat maximum.
The Flat Maximum
This is a general recommendation to read the book. I will also highlight a point Finlay makes. Namely about the existence of the flat maximum effect. According to Finlay “The flat maximum effect states that for most problems there is not a single best model that is substantially better than all others.” (Finlay, 2014, page 105).
What does this imply? That although the benefits to be gained from using analytics may often be significant. Any additional benefits can diminish relatively quickly after you already have a model and are simply searching for a better model. While some models may be a little better than some others the flat maximum effect means that you often don’t need to be too obsessive about using the perfect model. If one model isn’t necessarily the very best, but it is close and it works for your business, then you might choose the second best model.
Why Choose A Sub-Optimal Model?
One reason to choose a slightly sub-optimal model includes that this model seems credible. The fact that general, i.e. non-data-scientist, managers make it more likely to be implemented. Being likely to be implemented will make any model infinitely better than a supposedly superior model that isn’t likely to be used. (You can always run the superior model and compare the results to make sure you aren’t sacrificing too much).
The flat maximum effect helps explain why the perfect is often the enemy of the good. Why not accept good as it might be really quite near perfect and be actually achievable?
For more on analytics see here, here, and here.
Read: Steven Finlay (2014) Predictive Analytics, Data Mining and Big Data, Palgrave MacMillan