Rethinking the Weekly Scanner Data

Over 20 years ago, I led a minor revolt at IRI that eventually had CEO Gian Fulgoni make an executive decision. John LaRocca and Jim Sullivan had given me permission to write a position paper on why IRI should forego providing weekly data and move to delivering daily data. Efficient Marketing Services (EMS) was about four years old and led by a couple of ex-IRI executives and supported by one of the Middlegame heroes, Lee Cooper, that was mentioned in a previous blog. EMS was starting to be recognized as a strong competitor given its backing by Nielsen and was starting to deliver store-level daily information at the SKU level. We were hearing a lot about from our clients at P&G.

My principle idea was that we were getting analytically beat up since different retailer merchandising weeks had different start dates and none of them coincided with the IRI weeks ending on Sunday. By saving the daily data, IRI could recreate the sales units, actual prices and merchandising support in direct alignment with the retailer merchandising weeks. It would alleviate a lot of pressure we ran into with average prices that were often a mix of promoted and nonpromoted days of the week. A lot of work went into trying to make these numbers make sense when presenting to clients.

In the end, my proposal would overload the current systems. I remember being told that securing a census of stores among the retailers providing IRI with T-LOG data was a much bigger priority. I bought it and moved back onto ways to address the problem with the weekly data available. A lot of work has been done over the years at both IRI and Nielsen to deal with the issue, but there is also a lot going on with how retailers are utilizing price. Last year, an article in Great Britain predicted that technology would eventually even eliminate the weekly retailer merchandising calendar and further mask shopper choices relative to average price.

Great Britain has always been a fairly price-sensitive market and we often see this reflected in the relative elasticities and promotional lifts. Retailers know that better than Middlegame. Therefore, electronic price tags will start taking advantage of “peak time” pricing to account for the daily spikes in demand. These “e-prices” will automatically move the prices up and down on an hourly basis. According to the article, Tesco has already outfitted several of its stores and is testing the limits of pricing. Other retailers are also preparing “surge pricing” systems where store-specific prices rise and fall according to current demand for items in the store.

We have already seen these kinds of installations for supermarkets in the United States and Continental Europe. Middlegame has also heard clients complain that although they still invest in scanner-based price elasticity analyses, their commercial teams that interact with retailers repeatedly can’t use the work since none of the prices modelled are actually found by shoppers in the marketplace. It looks like if I really wanted to be some sort of futurist, I should have been asking for hourly data to be saved and not daily data all those years ago.

Regardless, this all points to the fact that we need to look at the problem differently. We either introduce new data that gives us that daily view and still run regression models like econometricians have done for more than half a century, or we can use the available data and different forms of models. Although I am greatly encouraged by new hourly data that will eventually be available from syndicated providers, Middlegame chose to use a different modelling approach. Since we search for cross-sectional variability over all the products in each combination of geography and timeframe, the relative price surges tend to cancel each other out. Relative across competitive products is the key and probably always will be.

Middlegame is the only ROMI consultancy of its kind that offers a holistic view of the implications of resource allocation and investment in the marketplace. Our approach to scenario-planning differs from other marketing analytics providers by addressing the anticipated outcome for every SKU (your portfolio and your competitors’) in every channel. Similar to the pieces in chess, each stakeholder can now evaluate the trade-offs of potential choices and collectively apply them to create win-win results.

Contact us at info@middlegame.com