For starters, we can split the data into high LTV and non-high LTV customers and analyze how well we recapture both labels. Accuracy over the last 6 months has been 89%. Meaning: 89% of predictions – high and non high LTV – we made 6 months ago proved to be correct over the last 6 months.
As far as the more fine-grained metrics go: Let's look at a comparison of what each customer actually spent over the past 6 months and what we predicted six months ago. The plot below shows the error our predictions made for every customer. Errors are small and centered around 0. The vast majority of the predictions (in mathematical terms, the 25th to the 75th percentile) fell between $0 - $17 The average error was $30 over the last 6 months, and the median error was $0.
When it comes to churn, customers Ocurate predicted to be most likely to churn six months ago actually churned in the ensuing 6 months. The below graph shows the number of customers that actually churned in the last 6 months and their model prediction six months ago. By far, most customers who actually churned, as measured in no new purchase during these six months, had a churn prediction of 95% or higher
