AI-Driven Personalisation: How Recommendation and Decision Systems Work

AI-driven personalisation is a decision system. It uses models and rules to choose or rank an eligible product, message, treatment or experience for a customer in a particular context.

The model is only one component. The delivered outcome may also depend on consent and permissions, product availability, contact policy, frequency caps, customer preferences, suppressions, commercial constraints, channel capability and fallback rules. A system can therefore use a sophisticated model and still make a poor decision, because the candidate set, objective, data, policy or experiment is wrong.

This guide explains how recommendation, propensity and next-best-action systems work, how to design their data and operating controls, and how to evaluate model quality, incremental business value, fairness and customer outcomes.

For the broader question of whether personalisation is the right AI use case in the first place, start with the wider AI-in-marketing framework.

What This Article Delivers

1. 1. The decision architecture Ten stages from eligible customer and candidate generation through features, scoring, constraints, delivery and logging.
2. 2. Recommendation approaches Collaborative filtering, content-based methods, hybrids, propensity, next-best action, bandits and session context, with the limits of each.
3. 3. Features and labels Why historical interaction data reflects the previous system's decisions, and how that quietly reproduces them.
4. 4. Cold start, drift and feedback What to do when there is no history, and how a recommender can homogenise behaviour without improving anything.
5. 5. Four layers of evaluation Offline metrics, online behaviour, commercial outcomes and guardrails, plus the experiment design that establishes incrementality.

Personalisation Is Not the Same as Segmentation

Segmentation assigns people or accounts to groups using shared characteristics. Personalisation changes a decision for an individual customer, account, device, household or session. The two combine readily: a segment may determine eligibility, while a model ranks the eligible options within it.

Personalisation is also not necessarily AI. Examples that require no learned model include:

• showing local store information from a postcode;
• suppressing a promotion after purchase;
• using a customer’s selected language;
• changing content for a known account type;
• applying a frequency cap;
• resuming an explicitly saved journey.

These rules may be preferable when the desired behaviour is clear, stable and easy to specify. AI-driven personalisation becomes relevant when the system estimates an outcome, learns a representation or adapts a ranking from data.

The Core Distinctions

Several concepts are treated as synonyms even though they answer different questions.

Concept	Question answered	What it does not establish
Eligibility	Is this action allowed and available?	Whether it is the best eligible action.
Recommendation	Which items appear relevant enough to consider?	Whether showing them causes incremental value.
Ranking	In what order should candidates appear?	Whether the list benefits the customer or the business.
Propensity	How likely is an outcome under observed conditions?	Whether a treatment changes that outcome.
Treatment responsiveness	For whom does one action cause a better outcome than another?	The customer's internal motivation.
Prediction	What outcome is estimated?	Why it happens, or what would happen under a different action.
Calibration	Do estimated probabilities or recommendation distributions match a defined reference?	That the selected treatment is incrementally valuable.
Real-time data	How recently was a signal processed?	That the signal is accurate, permitted or relevant.
Attribution	Which interaction receives credit under a rule or model?	The outcome that would have occurred without the interaction.
Incrementality	What changed because of the treatment?	Whether the effect will persist in another population or period.

A behavioural pattern is evidence of association in the observed data. It should not be described as a system understanding a customer or identifying an individual’s motivation.

The Architecture of an AI Personalisation Decision

A robust system can be understood as ten stages.

1. Define the Eligible Customer and Context

Establish who or what is being considered, and the decision moment. The context might be a website visit, an app session, a service interaction, a renewal window or a scheduled campaign.

Eligibility should be explicit. It may depend on:

• customer permission and communication preference;
• age or jurisdiction;
• current products or contracts;
• service status and vulnerability flags;
• channel availability;
• campaign or treatment rules;
• contact pressure and prior exposure;
• stock, price and fulfilment constraints.

Eligibility is a policy decision. It should not be left for the model to infer from historical absence.

Identity fragmentation can create duplicate profiles, incomplete histories and contradictory feature values. Every later stage inherits those errors: candidate generation, model training, eligibility and evaluation. Confirm how person, account, household, session and device identities are represented before introducing model-based decisions.

2. Generate Candidate Actions or Items

Candidate generation reduces the full catalogue of possible products, messages or actions to a manageable set. Candidates may come from:

• product or content categories;
• collaborative-filtering retrieval;
• content-based similarity;
• editorial or commercial collections;
• lifecycle rules;
• service actions;
• explicit customer requests;
• exploration pools;
• fallback and popular-item lists.

Large recommendation systems commonly separate retrieval from ranking. A first stage retrieves a relatively small candidate set, and a later stage scores those candidates using richer features. (1) (2) This separation allows speed, control and multiple sources of candidates.

Candidate quality sets a ceiling on the result. A ranker cannot select an appropriate item that never entered the candidate set.

3. Construct Permitted Features

Features are inputs available at the decision time. They can describe the customer, the item, the context or the relationship between them.

Examples include:

• prior purchases or content interactions;
• product attributes;
• session sequence;
• time since an event;
• channel and device;
• price or stock status;
• service history;
• frequency and recency of prior contact;
• aggregate popularity;
• a customer-stated preference.

Feature design must account for purpose, permission, minimisation, accuracy and availability at serving time. A value recorded after the outcome cannot legitimately be used to predict that outcome in production. Proxy variables can also reproduce sensitive or undesirable distinctions even when an explicit characteristic is absent.

A feature is not a fact about motivation. It is a measured or derived input that may help estimate a defined outcome.

4. Estimate Relevant Outcomes

A model may estimate:

• probability of click, purchase, renewal or cancellation;
• expected value, margin or watch time;
• relevance of an item to a customer;
• likelihood of accepting an action;
• expected service completion;
• treatment effect relative to another action.

The target must match the decision. Optimising clicks can favour attention-grabbing recommendations without improving completion, satisfaction or contribution. A published account of a large video platform’s production system describes changing the ranking objective from click probability towards expected watch time, because click-led ranking could promote content that was started but not completed. (2) That does not establish watch time as a universal objective. It illustrates why the chosen label shapes system behaviour.

5. Apply Policy, Eligibility and Safety Constraints

Before final selection, apply rules that the model must not override. These can include:

• product eligibility;
• legal or contractual restrictions;
• customer suppressions;
• stock and fulfilment;
• maximum contact frequency;
• price consistency;
• vulnerability controls;
• diversity or repetition limits;
• editorial exclusions;
• suitability and safety rules.

Rules and models are complements. A model can estimate relevance while rules enforce non-negotiable policy. Published practical guidance on recommendation systems describes re-ranking as the point at which freshness, diversity and fairness constraints can be applied. (1)

6. Rank or Select an Action

The system combines model scores and policy to produce a list or a single action. The final objective may be multi-dimensional, for example:

expected customer value - contact cost - risk penalty, subject to eligibility and guardrails.

The formula should be treated as a decision policy, not an objective truth. The weights reflect business choices. A fairness issue can arise from the model, from the candidate pool, from a commercial weighting, from a rule or from unequal access to the underlying product. Model fairness and business-policy fairness therefore need separate review.

7. Deliver the Experience

The selected action is rendered in a channel. Delivery can alter the result through:

• page position;
• creative execution;
• latency;
• device constraints;
• stock changes;
• message timing;
• channel failure;
• surrounding content.

AI personalisation owns which eligible item or treatment is selected. Dynamic creative optimisation owns modular creative assembly, paid-media delivery and creative-element testing. It can activate a decision, but it is not the decision architecture itself.

8. Log Exposure and Decision Context

Log enough information to reproduce and evaluate the decision:

• decision and customer or session identifier;
• timestamp;
• eligible candidates;
• features or feature-version references;
• model and policy version;
• scores;
• constraints applied;
• selected treatment and position;
• whether delivery succeeded;
• experiment assignment;
• human override;
• subsequent outcomes and their timing.

Do not log unnecessary personal data merely because storage is available. Logging should be designed around retention, access, correction and deletion requirements.

9. Measure Customer and Commercial Outcomes

Measure what happened after exposure and compare it with a credible counterfactual. Engagement may be an intermediate signal. The intended outcome may be completion, contribution, retention or reduced service effort. Guardrails may include complaints, returns, cancellations, opt-outs and subgroup disparities.

10. Update or Retrain Only Where Justified

A system should not retrain automatically merely because new data exist. First determine whether:

• the data distribution has changed;
• the relationship between features and outcomes has changed;
• product or policy constraints have changed;
• labels remain valid;
• experiment results support a new objective;
• retraining would reinforce an existing feedback loop;
• the change can be tested and rolled back.

Retraining is a controlled system change. It requires versioning, validation and monitoring.

Recommendation Approaches

No single method is universally superior. The right approach depends on available data, catalogue structure, latency, novelty, explainability and the cost of error.

Collaborative Filtering

Collaborative filtering uses patterns of interaction across users and items. It can identify items that tend to be consumed by similar users, or within related interaction histories, without requiring a complete hand-built description of each item.

These methods commonly represent users and items as embeddings in a shared lower-dimensional space learned from the interaction data. (1) More recent systems may learn those representations through neural networks, but the operational question is unchanged: do the learned relationships generalise to the live decision?

Useful when: there are enough interactions and shared behavioural patterns.

Limitations: sparse data, cold start, popularity bias, weak interpretability and feedback from prior recommendations.

Content-Based Recommendation

Content-based methods use attributes of items together with a representation of a user’s prior interests or current context. A content article can be represented by topic, format, author or embedding; a product by category, price, material or use.

Useful when: item metadata are strong, new items arrive frequently, or the catalogue contains specialised attributes.

Limitations: can over-specialise around previous interests, depends on metadata quality, and may miss useful cross-category relationships.

Hybrid Methods

Hybrid systems combine collaborative, content and contextual signals. They may generate candidates from several methods and use a common ranker, or blend scores from distinct models.

Useful when: different methods cover different failure modes.

Limitations: additional complexity, score comparability, and a heavier debugging and monitoring burden.

Propensity Models

A propensity model estimates the probability of an observed outcome, such as purchase or churn. It can prioritise review, or provide a feature for a decision system.

A high predicted purchase probability does not establish that a customer should receive an incentive. Customers who are most likely to purchase may be the least responsive to treatment, because they would have acted without it. The personalisation measurement framework covers the holdout designs needed to distinguish those outcomes.

Next-Best-Action Systems

A next-best-action system selects among eligible treatments, including the option to take no action. A mature system separates:

• eligibility and policy;
• predicted outcomes;
• treatment costs;
• customer and business value;
• constraints and guardrails;
• uncertainty;
• experimentation.

Best is always conditional on the stated objective and the available candidates. If the action set excludes service remedies and contains only sales offers, the system cannot discover that the most appropriate response is support.

Contextual Bandits

A contextual bandit selects an action using current context, while balancing exploitation of actions that appear to perform well against exploration that produces information about the alternatives. A foundational news-recommendation study framed article selection in this way and evaluated the method using randomised historical traffic. (3)

At an accessible level:

• exploitation chooses the action currently estimated to be strongest;
• exploration deliberately tries less certain actions in order to learn;
• the policy updates from observed rewards.

Bandits are not a licence for uncontrolled experimentation. Exploration must respect eligibility, safety, customer experience and exposure limits. A click reward can also teach the system to maximise clicks rather than customer value.

Sequence and Session Context

Order and timing can matter. A search, a product view and a checkout event are not interchangeable, and using future interactions to predict an earlier decision creates leakage. Sequence models or time-aware features can represent recent context, but they increase sensitivity to logging quality and to changing behaviour.

The same published account reports that label and context selection had a large effect on live performance, and warns that random hold-out approaches can leak future information where consumption is sequential. (2) That is a system-specific result, but the design lesson is general: construct training examples from the decision point that will exist in production.

Candidate Generation, Ranking and Re-ranking

The three stages solve different problems.

Candidate Generation

Optimise for coverage and retrieval: place plausible options in the set. Metrics may include recall of relevant items, catalogue coverage and retrieval latency.

Ranking

Assign a comparable score to candidates for the target decision. Metrics may include precision, recall, mean reciprocal rank, mean average precision or normalised discounted cumulative gain, depending on the task.

Re-ranking

Apply list-level and policy objectives that are difficult to express in a pointwise score, such as:

• diversity across categories;
• novelty;
• freshness;
• inventory or margin constraints;
• suppression of repeated items;
• fairness or exposure constraints;
• editorial balance.

A higher ranking metric does not guarantee commercial incrementality. It says that a list better matches the chosen relevance labels or the offline evaluation protocol.

Feature Design and Training Labels

Model behaviour follows from the examples it is asked to reproduce.

Features Should Be Available at Decision Time

Check for training-serving skew, late-arriving data and values that are populated differently across channels. Use the same feature definitions in training and serving wherever possible.

Labels Should Represent the Intended Outcome

A click is easy to observe but may be a weak label for satisfaction. Purchase is commercially relevant but sparse, and affected by price, availability and prior marketing. Retention is valuable but slow to observe and influenced by service quality.

Exposure Affects Labels

A customer cannot click an item that was not shown. Historical interaction data therefore reflect previous eligibility, ranking, creative and position. Treating a missing interaction as a neutral negative example can reproduce prior exposure decisions.

Negative Examples Need Definition

No purchase may mean no interest, no stock, no opportunity, an unsuitable price, a failed delivery or insufficient observation time. Label windows and censoring rules should be explicit.

Proxy Objectives Can Distort Behaviour

The model may achieve the proxy while undermining the real objective. This is why online outcomes and guardrails must accompany offline model metrics.

Real-Time Versus Batch Personalisation

The fastest possible decision is not always the best architecture.

Batch Decisions

Scores or rankings are calculated periodically and stored for use. This can suit renewal prioritisation, weekly recommendations or planned campaigns.

Advantages: simpler infrastructure, reproducible outputs, lower serving cost and easier approval.

Limitations: slower response to session context or inventory changes.

Real-Time Decisions

Features and scores are calculated at or near the point of interaction.

Advantages: can incorporate current context, availability and recent events.

Limitations: latency, identity resolution, missing data, serving reliability, a greater monitoring burden and more complex rollback.

A hybrid approach often works: batch candidate or propensity scores combined with real-time eligibility and context. Choose latency from the decision requirement, not from a general ambition to be real time.

Back in 2017 I went through several analytics accounts for a consumer electronics retailer and found something that needed no model at all. Desktop traffic peaked during working hours. Mobile traffic peaked between five and eight in the evening. We moved email and desktop web push to the mornings, and app push and mobile web push to around six. Click-through rates improved substantially, although I do not have a defensible number I am willing to publish for it. That decision needed a schedule, not a real-time system.

Cold Start

Cold start occurs when the system lacks interaction history for a new user, a new item or a new context.

New-User Cold Start

Possible responses include:

• ask for explicit preferences;
• use current-session context;
• use permitted broad priors;
• offer popular or editorially selected items;
• diversify early recommendations;
• reduce confidence and avoid high-cost actions.

New-Item Cold Start

Possible responses include:

• use product or content attributes;
• place new items in controlled exploration;
• use editorial tagging;
• create a dedicated candidate source;
• avoid treating absence of interaction as evidence of poor quality. (2)

New-Context Cold Start

A model trained on one channel, market or product may not transfer. Begin with conservative policies and test before assuming that learned relationships generalise.

This is the failure I would point to first. In fashion, making it easier to add to basket and sending people who arrived from an advertisement straight to a category page worked well. We tried the same treatment for a French automotive brand and bounce rates went up slightly. Fashion buyers want to reach the basket. Car buyers want to read and compare. We stopped the campaign and spent the effort on making product detail pages easier to read instead, which improved both bounce and overall engagement.

Cold start is not solved merely by collecting more data. The system needs a policy for uncertainty, and a way to learn without overexposing customers to poor options.

Exploration, Exploitation and Uncertainty

A system that always displays the current top-scoring item may never learn about the alternatives. It can become locked into early popularity and historical exposure patterns.

Controlled exploration can improve learning, but its purpose and boundaries should be explicit:

• which customers and items may enter exploration;
• maximum exposure and risk;
• reward definition;
• duration;
• how randomisation is logged;
• when an option leaves exploration;
• how harm or poor experience is detected.

Uncertainty should influence action. A low-confidence model may abstain, use a safe fallback or route the decision for human review. Presenting every score as equally certain encourages overconfident use.

Calibration of Predictive Scores

For a calibrated propensity model, cases assigned a probability near 0.2 should produce the outcome about 20 per cent of the time in the relevant evaluation setting. Discrimination metrics such as area under the ROC curve do not by themselves show that probabilities are reliable. Research comparing supervised-learning methods demonstrates that models with similar ranking ability can produce differently calibrated probabilities, and that calibration methods need independent data. (4)

Calibration matters when scores drive thresholds, budgets, expected-value calculations or capacity planning. Check it on later time periods and on relevant subgroups, because a score can be calibrated in aggregate while being unreliable for a particular population or after conditions change.

Calibration does not establish treatment responsiveness. A reliable estimate of purchase probability still does not show that a promotion causes an incremental purchase.

Popularity Bias, Feedback Loops and Filter Effects

Historical interactions are produced by the previous system. Items placed prominently receive more exposure, and therefore more opportunities to generate positive labels.

This can create a loop:

1. popular items receive more exposure;
2. exposure produces more interactions;
3. interactions strengthen their score;
4. the system gives them still more exposure.

Simulation research has shown that training recommender systems on behaviour already influenced by recommendations can homogenise behaviour without improving utility. (5) The result should not be treated as a universal prediction about every system, but it establishes a credible failure mechanism.

Mitigations include:

• logging propensities and randomised exposure where appropriate;
• separating organic and recommendation-driven interactions;
• maintaining non-model candidate sources;
• monitoring catalogue and supplier coverage;
• re-ranking for diversity and novelty;
• limiting repeated exposure;
• testing long-term outcomes;
• giving customers meaningful controls.

Diversity and novelty are not substitutes for relevance, but accuracy alone is also incomplete. Research on recommendation evaluation treats choice, discovery, relevance and rank position as distinct dimensions. (6)

Filter bubble should not be used as a catch-all label. Define the measurable concern instead: reduced category coverage, repeated exposure, supplier concentration, narrowing of content, a lack of new-item discovery or an identified customer harm.

Model Drift and System Drift

Drift can occur when:

• customer behaviour changes;
• the catalogue changes;
• pricing or stock changes;
• acquisition channels change;
• a campaign creates a temporary pattern;
• tracking or identity resolution changes;
• policy changes alter eligibility;
• the model changes customer behaviour and then learns from it.

Monitor more than aggregate accuracy. Useful indicators include:

• feature distributions and missingness;
• score distributions;
• calibration by period;
• candidate and catalogue coverage;
• subgroup errors and outcomes;
• override and fallback rates;
• latency and delivery failure;
• business and guardrail metrics;
• changes in exposure concentration.

A drift alert should lead to diagnosis, not to automatic retraining. The cause may be a broken feed, a changed product strategy or an invalid label rather than stale parameters. NIST’s AI implementation material recommends monitoring performance and trustworthiness over time, and recognises that business rules can restrict model outputs. (7)

Fairness, Customer Harm and Policy

Fairness has several layers:

• data fairness: whose behaviour and outcomes are represented;
• model performance: whether errors or calibration differ by relevant group;
• exposure fairness: who receives which opportunities, prices, products or content;
• policy fairness: whether eligibility and commercial rules create unjustified differences;
• outcome fairness: whether the full system produces unequal benefit or harm.

A model can have similar accuracy across groups while the underlying policy is unfair. Conversely, a group difference in a technical metric does not by itself establish unlawful discrimination or customer harm. The context, the affected decision, the legal basis and the consequences all matter.

Review:

• target and proxy variables;
• excluded populations;
• error and calibration by relevant subgroup;
• exposure and treatment rates;
• access to beneficial offers or service actions;
• complaint, cancellation and opt-out patterns;
• the feasibility of appeal, correction or human review.

The ICO distinguishes data-protection fairness, algorithmic fairness, bias and discrimination, and highlights how problem formulation, proxies and target variables can affect outcomes. (8) As at 28 July 2026, that guidance is under review following the Data (Use and Access) Act. Detailed organisation-wide governance and current AI-law classification belong in the ethics of AI in marketing.

For feature permissions, minimisation and resilient design, see privacy-safe personalisation and ethical consumer-data use.

How to Evaluate an AI Personalisation System

No single metric can establish success. Use four layers.

Offline Model Metrics

Metric family	Examples	What it helps assess	Main caution
Classification or retrieval	Precision, recall	Whether relevant cases or items are identified in the labelled dataset.	Depends on label quality and exposure history.
Ranking	Mean reciprocal rank, mean average precision, NDCG	Whether relevant items appear near the top.	Better rank may not increase incremental value.
Probability quality	Log loss, Brier score, calibration curve	Whether predicted probabilities are useful and reliable.	Must be checked over time and by relevant subgroup.
Coverage	User, item and catalogue coverage	Whether the system serves more than the easiest or most popular cases.	High coverage can include low-quality items.
Diversity and novelty	Intra-list diversity, category coverage, novelty	Whether lists avoid excessive repetition or concentration.	Definitions are context-dependent.
Subgroup performance	Error, calibration, coverage by group	Whether performance differs materially across populations.	Group choice and interpretation require legal and domain judgement.

Offline tests are useful for rejecting weak models and diagnosing behaviour. They are not final evidence of customer or commercial impact. Industrial recommendation research explicitly reports that live A/B results do not always correlate with offline experiments. (2)

Online Behavioural Metrics

Examples include:

• click-through rate;
• dwell or engagement;
• add-to-basket;
• completion;
• recommendation acceptance;
• task success;
• use of controls or negative feedback.

These reveal response to the delivered experience, including position and creative. They can still reward short-term attention rather than value.

Commercial Outcomes

Examples include:

• incremental conversion;
• contribution after discount, fulfilment and treatment cost;
• purchase frequency;
• retention;
• reduced avoidable service cost;
• customer lifetime value, where the model and observation period are defensible.

Commercial metrics need a comparator. Attributed revenue is not necessarily incremental revenue.

Guardrails

Examples include:

• complaints;
• opt-outs;
• suppression breaches;
• returns;
• cancellations;
• inappropriate recommendations;
• group disparities;
• customer-service contacts;
• repeated or excessive exposure;
• delivery and latency failures;
• unsafe or unavailable items.

Guardrails need thresholds and owners. Reporting them after a test without a response rule is not a control.

If you ask me which single loyalty or personalisation statistic misleads people most, my answer is that the question is wrong. Looking at any statistic on its own is what misleads. I sat through a great many client data reviews where a conversion rate uplift was presented as the result, and the average order value sitting behind it had moved the other way. The revenue picture was simply not in the number on the slide. This is a commercial observation about how results get reported, not a claim about any particular metric.

For a wider implementation of the measurement plan, use how to measure the effectiveness of personalisation and the personalisation KPI reference.

Why Better Offline Performance May Not Create Incremental Value

A model can improve an offline metric and still fail commercially, because:

• it predicts customers who would act without treatment;
• the treatment is weak, costly or poorly delivered;
• the candidate set excludes the best action;
• the label rewards the wrong behaviour;
• the model concentrates on already popular items;
• position or creative effects differ in production;
• the gain is too small relative to operating cost;
• the intervention causes fatigue, returns or cancellations;
• the training population differs from the live population.

The distinction between outcome prediction and treatment effect is fundamental. Causal-effect research notes that ordinary supervised learning predicts observed outcomes, whereas individual treatment effects involve an unobserved counterfactual: the same person cannot be observed both receiving and not receiving the treatment at the same moment. (9)

Use predictive models to estimate stated outcomes. Use randomised experiments, or another credible causal design, to test whether changing the decision produces incremental benefit.

Experimentation for Personalisation

Define the Treatment Precisely

Do not test AI against no AI if several things change at once. Specify:

• candidate sources;
• eligibility and constraints;
• ranking policy;
• creative or presentation;
• channel and timing;
• fallback behaviour.

Choose the Correct Unit of Randomisation

Randomise by customer, account, session, geography or another unit that limits contamination. Avoid switching a customer between incompatible policies where carry-over is likely.

Preserve Decision Logging

Record assignment, eligible options, selected treatment, version and delivery. Without exposure data, treatment effects and failures cannot be diagnosed.

Predefine Metrics and Duration

Set a primary outcome, guardrails, minimum detectable effect, sample assumptions and an observation window that captures delayed outcomes. Account for novelty and seasonality.

Test the Policy, Not Only the Model

The experiment should evaluate the complete decision system: model, rules, candidates, delivery and customer response.

Separate Ranking Quality From Incrementality

A ranker can select items that are likely to be clicked, while an experiment shows that customers would have clicked something else anyway. Report the two questions separately.

Watch for Interference

Inventory, social effects, shared accounts and marketplace supply can cause one participant’s treatment to affect another’s outcome. Note these limitations rather than treating every A/B test as automatically unbiased.

Keep a Stable Control or Holdout Where Justified

Longer-term holdouts can reveal fatigue, retention effects and drift, although they carry opportunity costs and need governance.

A recurring operational mistake is to treat a single winning experiment as a permanent result. User behaviour and audience composition both change underneath a result that was valid when it was measured. Maintain an appropriate persistent control where feasible, or define a revalidation cadence based on decision frequency, drift risk, seasonality, customer harm and the cost of continuing with a degraded policy.

Large advertising experiments demonstrate that incremental effects can be difficult to measure when outcome variance is high, even with large samples. (10) A non-significant result may therefore mean that the effect is small, that the test is underpowered, or that the measurement is noisy. It is not evidence that a preferred narrative is true.

Operating the System

A personalisation system needs ongoing product and operational ownership.

Daily or Automated Controls

• data feed completeness;
• identity and feature freshness;
• model-serving availability;
• score and candidate distributions;
• constraint and suppression breaches;
• delivery latency;
• fallback rates;
• experiment assignment integrity.

Periodic Review

• calibration and ranking metrics;
• subgroup performance;
• catalogue and exposure concentration;
• complaints, opt-outs and inappropriate outcomes;
• incremental results;
• total operating cost;
• vendor or model updates;
• policy and legal changes;
• continued need for the use case.

Change Control

Treat material changes to labels, objectives, features, candidate sources, rules and model versions as product changes. Validate them before full deployment and retain a rollback path.

Human Roles

Assign owners for:

• objective and treatment policy;
• data quality and permissions;
• model performance;
• experimentation;
• customer safeguards;
• incidents and complaints;
• pause and retirement decisions.

The model should not be the place where unresolved commercial, legal and customer-policy choices are hidden.

An Implementation Checklist

Problem and Policy

• What decision is being personalised?
• Which actions, including no action, are available?
• Who is eligible, and which rules are non-negotiable?
• What customer outcome should improve?

Data and Labels

• Which features are available at the decision time?
• Are they permitted, accurate enough and necessary?
• How were labels affected by previous exposure?
• Are outcomes observed within a useful period?

Model and Candidates

• Which candidate sources are used?
• What is the simplest baseline?
• Which objective is estimated?
• How are uncertainty and cold start handled?

Delivery and Logging

• Can the channel apply the decision reliably?
• Are model, policy, candidates and exposures reproducible?
• Is there a safe fallback?

Evaluation

• Which offline, online, commercial and guardrail metrics apply?
• What experiment estimates incrementality?
• Which subgroup and long-term checks are required?

Operations

• Who monitors drift and incidents?
• Which changes require revalidation?
• What triggers pause, rollback or retirement?

Conclusion

AI-driven personalisation is not a model that knows the customer. It is a governed decision process that combines candidates, features, estimates, rules, constraints, delivery and measurement.

The most important design choices often sit outside the algorithm: what is eligible, which outcomes count, how historical exposure shapes the labels, what the system is allowed to optimise, and how customer harm is detected. Offline accuracy can help build the system, but only online experiments and guardrails can show whether the complete policy creates incremental value without unacceptable effects.

For the prior decision about how to select an AI marketing use case, use the general pillar. For detailed creative assembly in paid media, use the dynamic creative optimisation guide rather than expanding this page into a creative-production workflow.

References

1. Google for Developers. Recommendation systems overview, Candidate generation overview and Re-ranking. Official instructional documentation forming part of the Recommendation Systems course. No reference number. No original publication date shown; all three pages last updated 25 August 2025 UTC. Not under review or in draft. Not UK law and not a regulatory source: company technical documentation produced by an organisation that develops recommendation systems and sells machine-learning and cloud products, so it carries a commercial interest. https://developers.google.com/machine-learning/recommendation/overview/types and https://developers.google.com/machine-learning/recommendation/overview/candidate-generation and https://developers.google.com/machine-learning/recommendation/dnn/re-ranking

2. Paul Covington, Jay Adams and Emre Sargin, Google. Deep Neural Networks for YouTube Recommendations. Peer-reviewed conference paper, Proceedings of the 10th ACM Conference on Recommender Systems (RecSys 2016). DOI 10.1145/2959100.2959190; ACM ISBN 978-1-4503-4035-9/16/09. Published September 2016; no subsequent update. Not under review or in draft. Not UK law: an industry research paper describing the authors’ own commercial platform, with a corresponding interest in it. https://doi.org/10.1145/2959100.2959190

3. Lihong Li, Wei Chu, John Langford and Robert E. Schapire. A Contextual-Bandit Approach to Personalized News Article Recommendation. Peer-reviewed conference paper, Proceedings of the 19th International Conference on World Wide Web (WWW 2010). DOI 10.1145/1772690.1772758. Published March 2010; no subsequent update, although the paper later received a 2023 Test of Time Award. Not under review or in draft. Not UK law: an industrial research study using a major news portal’s own traffic, with a corresponding interest in it. https://doi.org/10.1145/1772690.1772758

4. Alexandru Niculescu-Mizil and Rich Caruana, Cornell University. Predicting Good Probabilities With Supervised Learning. Peer-reviewed conference paper, Proceedings of the 22nd International Conference on Machine Learning (ICML 2005). DOI 10.1145/1102351.1102430. Published 2005; no subsequent update. Not under review or in draft. Not UK law: academic research with no product-sales interest disclosed in the paper, evaluating older algorithm families on binary classification problems. https://doi.org/10.1145/1102351.1102430

5. Allison J. B. Chaney, Brandon M. Stewart and Barbara E. Engelhardt. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. Peer-reviewed conference paper, Proceedings of the 12th ACM Conference on Recommender Systems (RecSys 2018), pages 224 to 232. DOI 10.1145/3240323.3240370. Published 2018; no subsequent update. Not under review or in draft. Not UK law: academic simulation research demonstrating a mechanism rather than measuring its size in any deployed system. https://doi.org/10.1145/3240323.3240370

6. Saúl Vargas and Pablo Castells. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. Peer-reviewed conference paper, Proceedings of the 5th ACM Conference on Recommender Systems (RecSys 2011). DOI 10.1145/2043932.2043955; ACM ISBN 978-1-4503-0683-6. Published 23 to 27 October 2011; no subsequent update. Not under review or in draft. Not UK law: academic research proposing a metric framework, with no product-sales interest stated. https://doi.org/10.1145/2043932.2043955

7. National Institute of Standards and Technology (NIST), US Department of Commerce. Manage, a page of the official AI Risk Management Framework Playbook, used together with Artificial Intelligence Risk Management Framework (AI RMF 1.0). Framework document and companion playbook. Reference number NIST AI 100-1; DOI 10.6028/NIST.AI.100-1. AI RMF 1.0 published January 2023; the Playbook page shows no publication or last-updated date and was checked on 28 July 2026. Under review: the live Playbook states that the AI RMF 1.0 is being updated and that the Playbook will be updated after the AI RMF is revised. No AI RMF 2.0 had been issued as at 28 July 2026. Not UK law: voluntary US federal guidance, sector-agnostic and not specific to recommender systems. https://airc.nist.gov/airmf-resources/playbook/manage/ and https://doi.org/10.6028/NIST.AI.100-1

8. Information Commissioner’s Office (ICO). Guidance on AI and data protection. UK regulator guidance; not a statutory code of practice. No reference number shown on the landing page. Updated 15 March 2023; status checked 28 July 2026. Under review: the live page states that, due to changes made by the Data (Use and Access) Act, the guidance is under review and may be subject to change. UK source and the applicable regulator for UK data protection. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/

9. Susan Athey and Guido Imbens. Recursive Partitioning for Heterogeneous Causal Effects. Peer-reviewed journal article, Proceedings of the National Academy of Sciences, volume 113, issue 27, pages 7353 to 7360. DOI 10.1073/pnas.1510489113; preprint arXiv:1504.01132. Preprint first submitted 5 April 2015; issue dated 5 July 2016; no subsequent update. Not under review or in draft. Not UK law: a methodological paper in causal inference, whose valid application depends on experimental or defensible observational assumptions. https://doi.org/10.1073/pnas.1510489113

10. Randall A. Lewis and Justin M. Rao. The Unfavorable Economics of Measuring the Returns to Advertising. Peer-reviewed journal article, The Quarterly Journal of Economics, volume 130, issue 4, pages 1941 to 1973. DOI 10.1093/qje/qjv023. Published online 6 July 2015; issue dated November 2015; no subsequent update. Not under review or in draft. Not UK law: an academic study of US digital-advertising field experiments with high outcome variance. https://doi.org/10.1093/qje/qjv023

Position as at 28 July 2026. Regulatory guidance and technical standards change. Reference 7 is expressly being revised and reference 8 is expressly under review following the Data (Use and Access) Act, so both should be rechecked before any decision is taken on the basis of this article.

Frequently Asked Questions