Day 6 — Wrap-Up

AI & Marketing

Simon Blanchard

This Lecture

  • Part 1: What This Course Was Trying to Be
  • Part 2: The Framework, Applied
  • Part 3: The Production-to-Orchestration Shift
  • Part 4: Evaluating AI Claims
  • Part 5: Exam Logistics

A Note on the Exam

The exam is May 6. The study guide is posted.

We will handle format questions at the end of class.

A Course on AI in Marketing Can Be Many Things

It could be a course on how to use AI tools — how to prompt ChatGPT, how to automate workflows, which features are in which product.

It could be a course on the frontier — agents, multimodal systems, whatever is trending this semester.

It could be a course on AI as a producer of marketing work — where AI generates the deliverable and you review it.

It could be a course on the psychology of AI — how consumers and marketers react to AI and how to react because of it.

This course was none of those.

It was a course on how to make decisions about AI systems.

Evaluating them. Diagnosing them. Deciding whether to deploy them. Deciding what to do in markets where others have already deployed them.

That is a different skill than operating AI tools.

This Lecture

  • Part 1: What This Course Was Trying to Be
  • Part 2: The Framework, Applied
  • Part 3: The Production-to-Orchestration Shift
  • Part 4: Evaluating AI Claims
  • Part 5: Exam Logistics

The Framework, From Day 1

From a business perspective, AI is not defined by whether a machine “thinks.” Or if it generates new knowledge.

It is defined by whether a system produces intelligent behavior through an engineering pipeline from input to action.

Six stages:

Stage What it is
Perception Raw input from the environment
Representation How the input is encoded
Model The function that operates on the representation
Algorithm The procedure the model uses
Constraints Rules that limit what the system can do
Behavior What the system does in the world

The claim from Day 1:

Every AI system we study in this course decomposes into these stages. Same stages. Different realizations.

The Framework Across Three Cases

Stage GenAI Chatbot Artea PittaRosso
Perception Text from user, structured data from database Experimental assignment + customer record Historical transactions, inventory, visits
Representation Embedded tokens; context window assembled from prompt + retrieved docs + history Behavioral variables (cart, channel, history) 8 product groups, 1 aggregated store, demand model
Model Retrieval-augmented context assembly around the generation engine Response prediction trained to target revenue Optimization
Algorithm The LLM: next-token prediction via self-attention ML fit to behavioral data Solver for optimization
Constraints System prompts, guardrails, API authorization None Revenue, margin, and sell-through targets
Behavior Generated responses, tool calls, CRM updates Send or withhold coupon Price and promotion recommendations

Case 1 — GenAI Chatbot

Framework

Perception. Text from user; structured data pulled from CRM and product databases at runtime.

Representation. Embedded tokens. The context window is assembled on every turn from system prompt, retrieved passages, and conversation history.

Model. Retrieval-augmented context assembly around the generation engine. The LLM is what the model uses; it is not the model.

Algorithm. The LLM. Next-token prediction via self-attention.

Constraints. System prompts, guardrails, API authorization.

Behavior. Generated responses, tool calls, CRM updates.

Lessons

A chatbot is a system, not a model.

The LLM is one component. A deployed chatbot also needs a retrieval layer for grounding, guardrails for policy, tool calls into the CRM, and logging for oversight. The hard part of building one is not picking the LLM. It is getting these pieces to work together reliably.

Chatbot economics are fragile.

HubSpot’s numbers looked decisive: $10,000 per acquired customer with humans, $7,000 with bots. Once CLV enters the picture, the case gets narrower. A 0.5 percentage point drop in retention, or a modest margin haircut, erases the savings. The acquisition number is visible. The retention and margin effects are not. The deployment case can look obvious and still break on numbers that were never in the pitch.

Case 2 — Artea

Framework

Perception. Experimental assignment and customer record.

Representation. Behavioral variables — cart contents, channel, purchase history.

Model. Response prediction trained to target revenue.

Algorithm. ML fit to behavioral data.

Constraints. None.

Behavior. Send or withhold the coupon.

Lessons

Ignoring a variable does not make a model unbiased.

The Artea policies never read gender or race. The behavioral variables they used carried demographic information anyway, through correlation, and no fairness constraint was applied to the output to prevent it. A model can be built without a protected attribute and still produce disparate outcomes.

Whether a model is biased depends on what you mean by biased.

The course introduced three fairness criteria — independence, separation, and sufficiency. They are not interchangeable, and when base rates differ across groups, they cannot all hold at once. Which criterion matters is a choice. A policy that looks fair on one criterion can fail another.

Prediction is not incremental value.

A model that predicts who will convert is not a policy that generates incremental revenue. The predicted-to-convert might have converted anyway. Predictive accuracy and policy value are different quantities, and an experiment is the only way to separate them.

Case 3 — PittaRosso

Framework

Perception. Historical transactions, inventory counts, store visits.

Representation. 8 product groups, 200 stores aggregated into one, and the demand model fitted from historical data.

Model. Optimization.

Algorithm. Solver.

Constraints. Revenue, margin, and sell-through targets.

Behavior. Price and promotion recommendations.

Lessons

A statistical model carries the regime it was estimated in.

PittaRosso’s demand model was fit on historical data in which every shoe was always discounted. When the strategy changed, those coefficients no longer described the world — but the system kept using them. This problem generalizes to any statistical model deployed outside the regime it was trained in.

Representation choices discard information that can later matter.

39,000 products were compressed into 8 groups, and 200 stores were treated as one. The aggregation hid size-level stockouts inside stores, so the system kept recommending against inventory that was no longer viable at the size level.

An incomplete objective produces a complete failure.

The original system optimized sell-through alone and discounted products that would have sold at full price. Adding revenue and margin targets fixed the symptom. The general pattern remains: any AI system scored on one dimension sacrifices the others. You need to put in the constraints.

This Lecture

  • Part 1: What This Course Was Trying to Be
  • Part 2: The Framework, Applied
  • Part 3: The Production-to-Orchestration Shift
  • Part 4: Evaluating AI Claims
  • Part 5: Exam Logistics

Remember this from day 1? AI Is Restructuring Marketing Jobs

Generative AI is not just changing how marketing work is done — it is changing which jobs exist.

Using 40 million U.S. job postings before and after the launch of ChatGPT, Luo, Miao, and Sudhir (2025) document a systematic production-to-orchestration shift:

  • Work that draws on bounded, retrievable knowledge (writing copy, handling inquiries, running reports) is being automated.
  • Work that requires judgment, coordination, and relationships (strategy, sales, account management) is expanding.

The bottleneck is moving — from producing artifacts to deciding what to do with them.

What happened to marketing employment after ChatGPT

Function Change
Content & Communications −17%
Channels & Support −9%
Research & Analytics −4%
Sales & CRM +13%
Strategy & Management +14%

And within the remaining jobs:

  • Entry-level roles: −14%
  • Executive roles: +9%
  • Average wages: +3.7% (despite fewer jobs)

Fewer jobs. Different jobs. Higher-paid jobs.

Source: Luo, Miao & Sudhir (2025), How Generative AI Reorganizes Knowledge Work: Evidence from 40 Million Jobs.

Every marketer now?

Mapping Course to Shift

What Luo calls orchestration work

Strategy. Decisions about what AI should do. Objective-function choice. Deployment timing.

Judgment. Evaluating whether a system is working. Reading a deployment and locating the failure.

Coordination. Working across marketing, analytics, legal, operations. Naming tradeoffs the data team cannot resolve alone.

Relationships. High-value customer work. Account-level decisions about where AI belongs.

What the course covered

→ AI is not just LLMs. The course covered supervised, unsupervised, and reinforcement learning, plus the generative model family. Different problems call for different types. Strategic decisions about which AI to use depend on knowing the options.

→ AI systems require integration, not just models. A deployed AI system is the model plus the data pipelines, retrieval layers, tool calls, and guardrails around it. Whether a firm has the infrastructure, the data, and the integrations to make it work is a separate question from whether the model exists.

→ Bias can enter without the variable. Behavioral features carry demographic correlations. Fairness criteria are not interchangeable, and choosing which criterion applies is a normative decision, not a technical one. Anticipating where bias enters is part of building the system, not something that can be added after.

→ Deployment decisions are economic decisions. Whether to deploy an AI system is a CLV question, not a technology question. Acquisition cost is the most visible input and rarely the decisive one. The economics have to be done with the model together.

This Lecture

  • Part 1: What This Course Was Trying to Be
  • Part 2: The Framework, Applied
  • Part 3: The Production-to-Orchestration Shift
  • Part 4: Evaluating AI Claims
  • Part 5: Exam Logistics

Evaluating AI Claims

The framework gives you a way to think about any AI system: what it perceives, how it represents that input, what it models, what it is constrained by, and what it does.

That is useful for diagnosing systems. It is also useful for something we have not yet named directly.

Evaluating claims about AI.

Practitioners will often present AI as transformational. Some of those claims are right. Many are not. The framework gives you a way to ask what is actually being measured, what the system is actually doing, and whether the claim matches the evidence.

Not every practitioner pushing a new tool is wrong.

But it is worth stopping before jumping on the bandwagon.

Ultimately the question is a business one, and AI is a tool that needs to be evaluated against business metrics.

What the Speaker Showed

Claim from the guest session:

AI search is changing everything for e-commerce.

Traditional search is giving way to AI tools. In the chart, the green portion — AI tools — has grown from under 10% in early 2023 to close to 40% by mid-2025.

If you listen to this story, the implication is clear: your firm should reallocate resources toward AI search now, before you fall behind.

That is a real claim, from a real practitioner, with real data.

Source: Semrush, guest presentation.

The Same Phenomenon, in Context

LLM share of total web traffic, by industry (Jan 2024 – May 2025).

The industries with the highest penetration:

  • Legal: 0.28% of sessions from LLMs
  • Finance: 0.24%
  • SMB: 0.22%
  • Health: 0.15%
  • Insurance: 0.11%

A fractional percent of all web traffic, even in the leading industries. The 527% growth headline is growth from a very small base.

Not nothing. Not transformational.

Source: Search Engine Land / Previsible, 2025.

searchengineland.com/ai-traffic-up-seo-rewritten-459954

And the Value Question

Kaiser, Schulze & Clement (2025)

Frontiers in Marketing Science.

973 e-commerce websites. About $20 billion in combined revenue. 50,000 ChatGPT-attributed transactions compared against 164 million transactions from traditional channels.

What they found:

  • LLM traffic is 200 times smaller than Google organic search, and less than 0.2% of total traffic.
  • ChatGPT visitors are interested enough to stay on the site, but convert less and spend less than visitors from other channels.
  • Across nearly a thousand sites, ChatGPT referrals produced lower revenue per visit than every major channel except paid social.

“oLLM currently serves niche informational needs of proficient consumers, not a broad conversion channel.”

Source: Kaiser, Schulze, & Clement (2025), Frontiers in Marketing Science.

The Takeaway

Both sources are looking at real data.

The speaker measures how much search activity has shifted toward AI tools. That share is growing quickly.

Kaiser measures what that traffic is worth in revenue per session. That value is low relative to every major channel except paid social.

A resource allocation decision needs both measurements. Neither alone tells the full story.

Growth and value are different questions. A channel can be growing fast and still not be where the revenue is coming from.

The same framework that lets a team diagnose an AI system also lets them evaluate claims about one. Both come down to asking what the system is actually doing, with what inputs, under what constraints, and against what business metric.

AI is a tool. It gets evaluated the same way other tools get evaluated — against what the business is actually trying to do.

Now

This Lecture

  • Part 1: What This Course Was Trying to Be
  • Part 2: The Framework, Applied
  • Part 3: The Production-to-Orchestration Shift
  • Part 4: Evaluating AI Claims
  • Part 5: Exam Logistics

Exam Logistics

When and where

In-person, during the MSB final exam period.

Format

3 hours. Paper and pencil. Closed book.

What to bring

Pen or pencil. A calculator — standalone, not your phone.

Sections

  1. Multiple choice — 30 pts
  2. Matching — 17 pts
  3. Short answer — 30 pts
  4. Calculations — 23 pts

Total: 100 points.

What to expect

Multiple choice is unambiguous. One correct answer per question. No “most likely” or “most defensible” wording.

Matching asks you to tag scenarios to framework stages, architectural components, or fairness criteria.

Short answer is structured — cells, labeled blanks, or short lists. Not essays. The prompt tells you the format.

Calculations cover CLV and token cost. Formulas and inputs are provided. Show your work — partial credit for correct setup.

Thank You