increase in ratio of high LTV customers acquired
increase in gross profit
increase in LTV of newly acquired high LTV customers
prediction accuracy
eSalon, a company in the women’s beauty brand space, Ocurate is an ML-driven solution that makes customer lifetime value (LTV) actionable for consumer brands. We founded Ocurate in late July 2021 on the premise of being able to pinpoint high LTV customers with unprecedented accuracy. Early results were benchmarked against simulation, showing about 90% accuracy.
Now, we are deployed long enough now with our existing customers to test against actual data – what LTV did we predict half a year ago, and what gross profit did customers actually generate? Here, we are starting with eSalon, a company in the women's beauty industry, and one of our earliest clients.
For starters, we can split the data into high LTV and non-high LTV customers and analyze how well we predicted both. Accuracy over the last 6 months has been 96%. Meaning: 96% of predictions – high and non high LTV – we made 6 months ago were proven to be correct over the last 6 months.
As far as the more fine-grained metrics go: Let's look at a comparison of what each customer actually spent over the past 6 months and what we predicted six months ago. The plot below shows the error our predictions made for every customer. Errors are small and centered around 0. The vast majority of the predictions (in mathematical terms, the 25% to the 75th percentile) fell between 0 and 7 cents. The average error was $3.54 over the last 6 months, and the median error was 0. For comparison: The company’s average LTV per year (measured in gross profit) is $49.
When it comes to churn, customers Ocurate predicted to be most likely to churn six months ago actually churned in the ensuing 6 months. The below graph shows the number of customers that actually churned in the last 6 months by model prediction. By far, most customers who actually churned, as measured in having made no new purchase during these six months, had a churn prediction of 95% or higher.
Since eSalon built their acquisition efforts on a real-time LTV model, we have a good way to evaluate the performance of Ocurate’s solution – - comparing the percent of high LTV customers acquired before and after implementation of Ocurate’s state of the art acquisition model, as a fraction of all customers acquired. Framing this analysis as a percentage rather than a discrete number, we address issues such as seasonality. Hence this can be considered a good evaluation metric.
In May, June, July and August, the average proportion of high LTV customers out of all customers acquired was 14%. In September, October, November and December, the average proportion of high LTV customers out of all customers acquired was 28.5%. This is an increase of 14.5 ppt, or over 100 Figure 1: Ocurate solution shows large improvements in acquiring high LTV customers over time.
Assuming a 30% targeted growth rate by customer count, this would add ((89,379*.285*$70.34)+(89,379*.715*$47.78)) - ((89,379*.14*$73.21)+(89,379*.86*$45.83)) = $406,352.69 in gross profit.
Another way to look at the same data: we can just compare the average predicted annualized LTV of the top 20% of newly acquired customers in the months leading up to Ocurate’s deployment, and after deployment. In the 4 months prior to deployment, the average LTV of the top 20% of newly acquired customers is $62.84; in the 4 months post deployment, the average LTV of the top 20% newly acquired customers was $69.16, an increase of over 10%.
