Uncovering the Message in the Mess of Big Data

This week I’m focusing on research that I’ve co-authored with Xin Wang in Business Horizons. Our article aims to explain to managers how they can work out what the messages are in large amounts of data. The classic application is to analyze product reviews on Amazon or similar websites. The structured data — 5 stars or 2 stars — has value at the headline level. Sadly knowing a product is 3.8/5 doesn’t help the manager know what to do to improve the product. The manager really wants to dig into why consumer likes, or doesn’t like, the product and how they are thinking about the product more generally. To do this we analyze the text that goes with the ratings.

The technique we recommend to do this is fairly well know in computer science/statistics. This means managers have a decent chance of being able to hire talent (probably Masters level graduates) to implement the technique. “Latent Dirichlet Allocation (LDA) can analyze huge amounts of text and describe the content as focusing on unseen attributes in a specific weighting. For example, a review of a graphic novel might be analyzed to focus 70% on the storyline and 30% on the graphics.” (Bendle and Wang, 2016, page 115).

The idea is that we can understand what consumers’ care about for what they talk about. LDA can even be modified to assign a valance to topics — i.e., to show which topics are seen as positively or negatively by consumers. This is important where attributes that get mentioned a lot may be thought of as good or bad. If consumers talk about laptop speed are they complaining, or are they impressed?

Interestingly business strategists can use the same technique to better understand the market. As we saw the “.. value of this technique extends well beyond the CMO’s office as LDA can map the relative strategic positions of competitors where they matter most: in the minds of consumers.” (Bendle and Wang, 2016, page 115). We are living in a big data world but the good news is that there are techniques, like LDA, to help us understand the messages.

As a special bonus we have posted slides on the Business Horizons website which summarize the work. (Plus you get to hear my talents as a voiceover artist). Click here to listen.

Hope you enjoy it.

Read: Neil Bendle and Xin Wang (2016) Uncovering the message from the mess of big data, Business Horizons, Volume 59, Issue 1, January–February 2016, Pages 115–124