Tuesday 20 February 2018

DIY Volume Forecasts from concept tests



At the 2010 AMSRS conference, a paper was presented dealing with the difficulties of using purchase intent scores to estimate sales for a new product.

The claim was made we can’t simply take these scores, however much we may modify or recode them, and use them for this purpose.  So an approach was described, whereby the purchase intent scores for a similar/comparable product that had been launched were used to devise a calibration factor for the scores obtained for the test product.

That’s not a bad idea … however, it’s not quite so simple.

Purchase intent, properly considered, measures only the respondent’s likelihood that they will try the product.  There is no guarantee that, once having tried, the respondent would go on to adopt the product into his/her repertoire (i.e. keep on buying it).

Indeed, if the product turns out to be a ‘dog’, then there is very little likelihood that the respondent will ever purchase that product again.  [Why are dogs used to represent something bad?  I know some very nice dogs.]

So, properly considered, purchase intent scores are only one part of the estimation process.  This is why, some considerable number of years ago, “trial-repeat” modelling arose.  It’s an old approach, but I have yet to find one better.

The underlying premise of trial-repeat modelling is that the steady-state long-run market share achieved by a (new) product will be the product of the long-run levels of trial and repeat purchasing it attains.

Thus (slightly simplified):

            MR = TR * RR

                        where:            MR = steady-state long-run market share
                                              TR  = long-run cumulative trial rate
                                              RR  = long-run repeat purchase rate.

Further (again, slightly simplified):

  • Trial comes through initial purchase
  • Initial purchase depends on the level of awareness (brought about by advertising and promotion) and on the availability of the product in stores.
 
Thus:

TR = TP* A* D

where:            TP = long-run probability of trial
                               A  = long-run level of awareness
                               D  = long-run level of distribution.

All the above parameters are reasonably readily able to be estimated, except RR - the long-run repeat purchase rate.

TP can be estimated from the proportion of respondents who indicate at the initial interview of a product test that they would buy or choose the test product.

D and A can be specified based on experience or by some other means.

RR - the long-run repeat purchase rate – can actually be viewed as the equilibrium, or steady-state solution of a first-order, two-state Markov chain.

In the table below, R1 is the probability that someone would be in a state of not having bought the test product, and would move into a state of having bought the test product.  R2 is defined similarly.



Some people will stay permanently in the ‘not buy’ state and others will stay permanently in the ‘buy’ state.  And some will be in a constant state of flux, moving periodically from ‘not buy this time’ or ‘buy this time’ to ‘not buy next time’ or ‘buy next time’. 

In the long term, for a group of people it can be shown that mathematically things settle down to a steady state situation, RR, the long-run repeat purchase rate mentioned above.

In fact, RR turns out to be exactly computed as[1]:

      RR = R1  /  (R1 + 1 - R2).

Where do R1 and R2 come from?

In some simulated test market models, where respondents are interviewed both before and after the product is placed with them to try, estimates of R1 and R2 are based on the behaviour (or preferences) at the follow-up interview:

  • R1 is the proportion of those respondents who did not buy/choose the test product at the initial interview, who did buy/choose at the follow-up interview.
  • R2 is the proportion of those respondents who did buy/choose the test product at the initial interview, who did so again at the follow-up interview.

But if we are not actually undertaking a full-scale simulated test market modelling study, what do we do?

From a one stage product test (i.e. where product is not actually placed with respondents), we actually have some information in the answers obtained to the purchase intent questions.

We can assume, for example, that:

      “I would definitely buy this product” = 70% probability of purchase
      “I would probably buy this product” = 30% probability of purchase.

An estimate of R1 can then be computed directly from this information, perhaps with some additional tweaks.  R2 can take an assumed value, normally around 25% to 35% (depending on the category) and the trial-repeat calculations then worked through to give the long-run market share estimate MR.

In reality, we should not make fixed assumptions about any of the various parameters discussed above.  Ideally, we would conduct a number of Monte Carlo simulations, where values for each parameter are sampled from a specified distribution.  That’s not too hard to do, and it gives us the option of providing an estimated product sales figure that sits between an upper and lower bound, or (better) 25th, median and 75th percentile estimates (based on the estimated values for MR, in combination with population data, and weight and frequency of purchase).

My experience in doing this over very many projects is that the eventual actual sales often lie pretty much in the middle of that range, i.e. more or less coinciding with the median forecast.


[1] William Feller, An Introduction to Probability Theory and its Applications: Vol 1, 3rd edition, Wiley 1968.

Tuesday 15 August 2017

Not Precisely Stated?



The NPS® question is asked on a 0 thru 10 scale.  The so-called Net Promoter Score® is then calculated at an aggregate level (i.e. for the whole sample) as the difference between the percentage of respondents giving 9 or 10 as their answer less the percentage giving 0 thru 6 as their answer.

I make no comment as to the suitability or otherwise of such a metric – there’s plenty of discussion on the net which deals with that, plus two papers about to be presented on the topic at the forthcoming AMSRS 2017 Conference ...

https://www.amsrs.com.au/conference-information

But what many do not realise is that Net Promoter Score® is in effect a simple 3-point scale, which would on the surface not seem to have quite the same cachet as the 0 thru 10 scale that’s administered in surveys.

Why do I say that?  Because mathematically the Net Promoter Score® is equivalent to each respondent giving -100 (i.e 0 thru 6), 0 (i.e. 7 or 8) or +100 (i.e. 9 or 10) as their answer.  And that’s not roughly equivalent, it’s precisely equivalent.  If you calculate the average score across your sample using those values, you get exactly the same result as subtracting the overall percentage giving 0 thru 6 from the overall percentage giving 9 or 10.

So how do we feel about that, i.e a very lumpy 3-point scale?  Does it seem reasonable to measure customers’ feelings about their product/service supplier in that way? 

Surely we could just as easily ask them to give a score of -100, 0 or +100 with -100 = ‘wouldn’t recommend highly’, +100 = ‘would recommend highly’, and 0 = ‘might or might not recommend highly’, or somesuch.

Or we could ask them similarly to give a score of -1, 0 or +1 and rescale the result by a factor of 100.

I honestly don’t know how I feel about this.  I am frequently asked to analyse data to produce NPS® ratings, and I will no doubt be continuing to do this into the future.  But I think it behoves all of us to give some serious consideration to the advisability, or otherwise, of basing so much on what appears to be so little.

Wednesday 5 October 2016

How to tell someone 'thanks but no thanks' ...

From a current supplier of software to the market research industry, when a request was made to correct the way they compile dummy data to test the results from an online survey ...

"... we value our client input and are grateful you took the time to share your needs. Therefore, I can file a feature request in your behalf. Note that this doesn't guarantee that the feature will be implemented or placed on a timeline, but it does ensure that it will be available for future consideration."

In other words "Don't hold your breath waiting for us to do anything." 


Tuesday 3 May 2016

NPS again ...

Here's a rather good, and contemporary, dissection of the pros and cons of using NPS ...

http://www.beatonglobal.com/nps-is-no-panacea/




Tuesday 15 December 2015

ANZ's new website ... how to really annoy customers

It appears that ANZ has somehow got the idea that they know best ... did they do no useability testing at all?

Me (to ANZ):

Please please revert your website back to the old one. The new one is a disaster. It takes 3 times as long to do anything, because it's always "waiting for www.banking4.anz.com" For running my business and personal accounts, it's now pretty much unusable from a desktop PC. 


ANZ (to me):

We appreciate your feedback.
The new ANZ Internet Banking design was made to reflect the changing banking needs of our customers, and has a consistent feel across eligible desktop, tablet and mobile devices. We understand that you enjoyed the previous ANZ Internet Banking design and we hope that you will experience the benefits of our new design in time.
Unfortunately we are unable to revert customers to the previous ANZ Internet Banking design.
  


 

Friday 6 November 2015

Onedrive .... I've finally given in ...

Just letting you know (because I have been whinging about it for so long) that I've finally dumped Microsoft's Onedrive as my primary cloud storage facility, and signed up for a 1Tb Dropbox account.
Sure, Onedrive (which I still have access to, by virtue of my Office 365 subscription) gives me 10Tb for free, and Dropbox costs about $AUD120 per year, but at least it appears seamless, trouble-free and fast.


More good info here ... http://www.computerworld.com/article/2848118/microsofts-defense-of-onedrive-changes-fails-to-silence-critics.html

Monday 5 October 2015

Ho hum ... same, same

So I signed up my mum to the www.my.gov.au website, to 'simplify' dealing with her Centrelink and ATO requirements.  Hah.  Read the train of screenshots below, starting from the top.  How on earth they expect the elderly to cope with this sort of non-service is beyond me.











Then, on the Tuesday after the weekend, to give the ATO time to implement its maintenance, I get this message ...