Demand Forecasting for a Large Retail Client

A large retail client had recently implemented machine learning algorithms for demand forecasting, however, the performance was not meeting their goals. They needed support to diagnose the issues and implement improvements.

Challenges included:

• High variability in accuracy across categories.
• Uncertainty about the drivers of forecast misses.
• An underdeveloped process for implementing & testing algorithm improvements.

With dozens of categories performing below target, their team needed extra hands on-deck to dive in and sort out the situation.

How we helped the client

In the first phase of the project, we assessed the client's data quality and their processes for measuring accuracy. We found a number of bugs and data quality issues that were driving unexpected measurements. In addition, we found that the team was generally testing on a single 12-week period, resulting in accuracy measurements that did not reflect seasonal variation throughout the year.

In the second phase of the project, we improved their metrics & measurement code, and developed a pipeline for multi-period time series cross-validation. We discovered that the team's single test-period approach was primarily motivated by slow run times. We refactored their code with polars and implemented a number of other efficiencies that allowed us reduce run times by 70+%.

In the third phase of the project, we implemented new features, changed the model fit grain, and implemented a hyperparameter tuning and feature selection pipeline. These changes resulting in substantial accuracy improvements across all categories (~25% improvement). In addition, we developed a roadmap for future improvements.

Results

By the end of the project the client had a clear understanding of their data, a robust set of tools for testing & measurement, and substantially more accurate models in production. Key benefits included:

• Improved Accuracy: ~25% improvement in accuracy across all categories.
• Faster Run Times: a ~70% reduction in run times, allowing them to iterate rapidly and identify effective changes faster.
• A Clear Roadmap for the Future: a clear roadmap for future improvements based on a broad review of forecasting best-practices and the changes found to be effective during the project.

keywords:

[Accuracy Improvement: 25+%][Run Time Reduction: 70+%][]