Steven Finlay has a useful book on Data Science, (Predictive Analytics, Data Mining and Big Data). He has lots of helpful practical advice in an easy to access form. Beyond a general recommendation to read the book I will highlight a point he makes — namely the existence of the flat maximum effect. According to Finlay “The flat maximum effect states that for most problems there is not a single best model that is substantially better than all others.” (Finlay, 2014, page 105).
This means that although the benefits to be gained from using analytics may often be significant they can diminish relatively quickly after you already have a model and are simply searching for a better model. While some models may be a little better than some others the flat maximum effect means that you often don’t need to be too obsessive about using the perfect model. If one model isn’t necessarily the very best, but it is close and it works for your business, then you might choose the second best model. One of the reasons to choose a slightly sub-optimal model includes that this model seems credible to the general, i.e. non-data-scientist, managers and so is more likely to be implemented. This will make it infinitely better than a supposedly superior model that isn’t likely to be used. (You can always run the superior model and compare the results to make sure you aren’t sacrificing too much).
The flat maximum effect helps explain why the perfect is often the enemy of the good. Why not accept good as it might be really quite near perfect and be actually achievable?
Read: Steven Finlay (2014) Predictive Analytics, Data Mining and Big Data, Palgrave MacMillan