Summary
Traditional, correlational-based AI approaches often fail to make an impact when allocating resources to pricing and promotion
- they struggle to de-confound the relationships between features, making it difficult to isolate the exact effect of a pricing/ promotion decision
- they lack explainability & trust from category managers
decisionOS by causaLens empowers data scientists to create decision workflows for pricing and promotion teams that
- suggests optimal interventions to maximize return on pricing and promotion investments
- are fully explainable through causal graphs and structural causal models
Background
Organizational investment in pricing and promotions continues to grow, yet 70% of initiatives deliver a negative return. At the same time, the most successful initiatives offer excellent returns. This shows the vast opportunity for retailers and CPGs if they can optimize their pricing and promotion strategies. As such, both retailers and CPGs are keen to make better decisions when it comes to allocating resources to pricing and promotions.
Building decision workflows which leverage organizational data is key to improving the pricing and promotion ROI. Creating these requires building effective measurement tools, to accurately understand the effects of past pricing and promotion campaigns, along with the ability to ask ‘what-if’ questions to decide how to allocate future resources. With decisionOS by causaLens, data scientists can quickly create and productionize decision workflows.
Why not traditional ML techniques?
This type of problem is inherently challenging to address with traditional machine learning techniques. The main reason for this is that it is very difficult to isolate the exact effect of a pricing/ promotion decision. A decision that appears successful or unsuccessful may be confounded by a wide range of factors:
- Competition: Current competitor pricing, competition levels around a particular store etc
- External factors: eg. time of year, weather, store location, external events etc
- Promotion type: How much is a product discounted by? What is the size of the discount? Is it a Buy One Get One Free offer?
- Surrounding internal factors: other promotions currently being run, pricing of substitutes or complements
With Causal AI it is possible to go beyond the traditional ML models by creating decision workflows which account for all potential effects and allow the true effect of taking a given action to be measured.
This article outlines how decisionOS by causaLens seamlessly facilitates the creation of models which accurately measure the success of pricing and promotion choices. Further, it will explain how this information can be used to inform future pricing and promotion decisions.
Specifically, this blog will focus on building a causal decision workflow which models the relationship between price and demand, and uses this to measure the success of two previous promotions by understanding their impact using counterfactual analysis.
Causal Workflow
To create decision workflows, an understanding of the overall causal workflow is required. The causal workflow consists of the following stages:
1. Data Preparation and Causal Feature Engineering: load data from a filestore & Find features that are causal towards the target
2. Human Guided Causal Discovery: Input domain knowledge (This can be done with tiers and/or edge constraints) & Use domain-knowledge and algorithmic causal discovery algorithms to find a causal graph which describes causal relationships between features
3. Causal Model Discovery: Given a causal graph, build a structural causal model
4. Causal Decision Intelligence Engines: Define counterfactual analysis (e.g., How promotions affect demand), Run counterfactual analyses & Use the outputs to inform future pricing and promotion decisions
Data Preparation and Causal Feature Selection
First, data needs to be loaded from the mounted file store.
Figure 1 shows how this works with a CSV file. decisionOS also contains the necessary connectors to integrate with a range of live data sources e.g., Snowflake, BigQuery, AWS S3, DynamoDB, Hadoop etc. Once this data has been loaded, additional data sources can be included, for example, macro indicators, Google trends and consumer confidence data.
Feature Discovery
As a range of data sources are being used here, features may need to be transformed to be meaningful to the target. Feature Discovery can quickly output the transforms with the most signal.
Once the feature transforms have taken place, it is important to understand which features are not only significant but causally related to the target. The causal feature selection part of the Causal Workflow can be used to find these causal drivers.
Causal Feature Selection
decisionOS provides access to a wide range of state-of-the-art Causal Discovery algorithms that separate correlated variables into causal drivers and confounded variables. Available algorithms include:
- FSNOD
- Multivariate Granger Causality
- PCMCI
- PCMCI AR
- PCMCI Ensemble
- PCMCI One
- PCMCI Zero
- sypi
- VAR LINGAM
- VAR NOTEARS
In this instance, PCMCI will be used to confirm the causal drivers of demand.
With the help of PCMCI causal feature selection, all of the features which are causal towards the target have been found. This knowledge can now be used along with human guided causal discovery to find a full causal graph.
Human-Guided Causal discovery
Human Guided Causal Discovery uses a combination of domain-knowledge from experts and algorithmic techniques to discover a full causal graph.
Expert domain knowledge can be provided in two different ways:
- Hierarchy of variables(Tiers): Tiers are defined in a hierarchy in descending order. Variables cannot affect variables above them in the hierarchy.
- Domain Knowledge: Allows relationships between variables to be defined and places where relationships are forbidden to be highligted
This is facilitated by the Human Guided Causal Discovery app provided by causaLens
Discovering a full graph
Once domain-knowledge has been added, causal discovery algorithms can be used to discover a full graph. The following algorithms are all available within decisionOS, many of them enhanced by the causaLens R&D team to improve performance:
- A*
- A* Global
- A* Local
- FCI
- FCI Exogenous
- FCI Probabilistic Tiers
- FCI Tiers
In this example, the FCI tiers algorithm will be used to discover the rest of the causal graph while accounting for defined tiers and domain knowledge. The domain knowledge defined in the Human Guided Causal Discovery app can be retrieved through an API.
Causal Model Discovery
To identify the effect of one node on another within the learned causal graph, the functional relationships (also known as the structural causal model) for the edges in the graph need to be learned. This creates an understanding of how features affect each other and the target. Additionally, this allows interventions on a specific node in the graph to be modeled and downstream effects understood.
The structural causal model is learned using causaLens’ proprietary causalNet algorithm. The input to this algorithm is the learned causal graph and the constrained relationships between the nodes, e.g., monotonically increasing/decreasing as determined in the causal discovery step.
causaLens also supports a novel decision tree algorithm (CLDT) and has interfaces to enable usage of open-source causal algorithms from packages such as PyWhy or EconML.
The causalNet algorithm will learn the optimal parameters to fit the observed data across all features to predict each of the nodes in the graph using its direct parents.
To obtain a good causalNet model, hyperparameters also need to be optimized. causalens’ model discovery experiment utilises double machine learning (doubleML) to find the best model for the given data based on a user defined configuration. doubleML is crucial in accurately finding price elasticity as both price and demand are confounded by a large number of shared drivers. This is done automatically by causalNet.
Once these steps are complete, the user has a structural causal model for understanding pricing and promotions. The next step is to find the optimal actions which should be taken using Decision intelligence engines.
Decision intelligence engines
Traditional correlation-based machine learning models can perform predictions. Causal models can be used to go beyond simple predictions, generating actionable business insights. This is achieved through various techniques such as interventional analysis, counterfactual analysis, effect estimation etc which are unique to Causal AI.
Causal decision intelligence engines are a collection of components which decisionOS exposes in a simple API and UI. This enables data scientists to plug in their data, causal graph and model to immediately generate actionable insights. The set of Causal Decision Intelligence Engines includes:
- Causal Fairness & Bias:Determine how discrimination can occur within your data as you perform interventions and actions
- Causal Effects:Simulate how interventions impact different groups within your data
- Root Cause Analysis: Understand the underlying reasons for why certain outcomes occur
- Algorithmic Recourse:Provide optimal interventions for a given objective
For the price and demand optimization, the following example shows how we can use causal effects: What would happen to my demand if my competitors launched certain promotional campaigns?
Promotion 1:
- Duration: 4 Weeks
- Products: 1L bottle Coke, 1L bottle 7-Up
- Locations: Michigan
- Discount: 30%
Promotion 2:
- Duration: 6 Weeks
- Products: 1 can Coke, 1 can 7-Up
- Locations: Houston
- Discount: 25%
These promotions can be defined simply in decisionOS and then causalNet can be used to process each promotion as a counterfactual. Intermediary nodes like Competitor Demand will be recalculated based on the learned structural causal model, as will the target – Demand.
Comparing demand levels
The original demand levels and the counterfactual demand for each of the given promotions can be compared to understand the success of different promotion types while controlling for a range of variables (e.g., time, location etc)
Once the underlying model and decision intelligence has been created, it can be simply packaged up into a decisionApp so that business stakeholders can interact with the modeling work. The decisionApp can be used to make pricing and promotion decisions based on live data and given business constraints e.g. what would be the optimal promotion have been at Halloween given my competitors ran a promotion of 15% on Pepsi.
decisionOS seamlessly facilitates the creation of decision workflows to optimize pricing and promotion decisions. Using Causal AI, it is possible to understand counterfactual demand and isolate the exact effects of given pricing/ promotion decisions, so that historical decisions can be accurately measured, and future decisions optimized. Organizations that use Causal AI for pricing and promotion see a significant increase in the number of successful campaigns run.
