Your metrics may be more arbitrary (and less useful) than you think.

By Andy Myers

In this era of data analytics and data science driven by machine learning (ML) and artificial intelligence (AI), there’s often a temptation to jump straight to solving the most complex business problems because these new techniques allow for it. But for most businesses, a more immediate and tangible opportunity exists.

The quick approach

Viewing performance through arbitrary bands may be hiding something that’s lying underneath. Perhaps your customers follow a nine-week purchase cycle? What if the thresholds for a specific spend behavior are $13 and $42? Would you ever be able to take action upon these insights based on reports using your existing, arbitrarily constructed (though easy-to-understand) bands?

“Eyeballing” data is great for sense-checking and spotting obvious errors, but when you’re planning to make decisions based upon it, it’s fraught with danger. You may miss trends that aren’t necessarily visible to the human eye. If you’re only looking at customer behavior by quartile — using a box plot, perhaps — would you notice a bimodal distribution?

To properly recalibrate your metrics and extract the most value from your analytical efforts, you’ll need:

A Stable Analysis Period

In other words, a stable time period should provide an appropriate snapshot of your business, be relatively quick to analyze and allow you to be confident in the results. For example, a department store might have a stable period of 12 months because most customers will likely shop at least once during that time. A grocery store, however, which most of us visit much more frequently than a department store, might have a stable period in the 6–8 week range.

By identifying this stable period, you can put analytical insights into action and measure their effects sooner. Running tests on this basis instead of using an arbitrary time period means you can plan projects more effectively. It also brings consistency to all the analysis across the business, from BI reporting to marketing measurement and custom analysis projects.

A view across multiple stable periods

· Seasonality effects: Q4 holiday sales spikes and Q1 sales lulls.

· Product performance: The popularity of a TV series-over-season.

· Market changes: A new product launch or product de-listing.

· Weather: Impacting consumers’ ability to visit your business.

Taking this approach, you can set consistent, insightful and actionable ranges to track metrics such as average spend, frequency or quantity. They’ll tell you what the natural range splits are for your metrics, and they’ll also allow you to quantify how much you need to shift behavior to make a meaningful difference. For example, encouraging customers to [visit/buy/watch] one more [time/item/show] will be difficult to execute if the natural frequency bands for your customer segment are too wide, because customer behavior within the band will be too diverse to steer effectively.

Moreover, defining customer segments using this approach means they’re likely to hold steady over time, providing a more robust framework by which to measure behavioral migration. For example, moving people up a value chain or identifying behavioral changes that lead to lapsing.

This analysis is conducted using migration clustering, which involves observing the distribution of your customers for key metrics over two consecutive stable periods, and hierarchically clustering those customers into groups based on the combination of the percentiles (or n-tile depending on the data) into which they fall in the two distributions. The ultimate goal is to identify the behaviors that are common and consistent period-over-period, and to ignore outlying, anomalous behaviors. The clusters account for some natural movement within themselves, meaning that movement between then should require some outside stimulus from the business, such as targeted marketing.

Pragmatic, thoughtful modeling

Selecting useful cluster sizes

This is the art of data modeling. It’s a subtle art — the bit that the computer can’t get to, the real-world context that the algorithm can’t include. A good data analyst or data scientist doesn’t just make these decisions by eyeballing or arbitrarily choosing cluster boundaries based on a whim. They’re combining the algorithm’s power with their own knowledge of how the business operates to fine-tune the solution for maximum utility by executives, marketers, and other stakeholders across the business.

This migration clustering technique can be used in isolation to set standard bands for metrics to feed into reports or for practical application. For example, using frequency or spend bands to create loyalty tiers or to display upgrade or free shipping threshold pricing on a website. They can also be combined to form behavioral-based customer segmentations. Consider overlaying spend and frequency bands and clustering the combinations to form a value segmentation matrix that can inform CRM decisions to optimize customer value.

RFM segmentation

Taking the time to set these up means that the rest of your analyses that use these bands — and these will likely include some of the key success metrics for your business — will be more robust, more stable and aid better decision-making overall. Not to mention elevating your own knowledge and evolving your intuition of how your customers are behaving.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aviary Analytics

We help clients achieve transformative growth using advanced analytics, data modeling, and AI.