Prime Video AI Support Chat: Reducing POM Dependency in LLM-Based RAG Chatbot
Every product team working at scale knows the quiet tax of POM dependency. Product Operations Managers field the same questions repeatedly, routing partners and stakeholders to the same documentation, answering the same edge cases, and manually triaging requests that a well-built chatbot could handle in seconds. The idea here was straightforward: build an AI-powered support chatbot for Prime Video's Slate POM workflow that could absorb that repetitive load, answer FAQ-level questions confidently, and hand off only the genuinely complex cases to a human. What followed was one of the more honest engineering and design journeys I have worked through, and it did not go the way I expected.
The Problem
In a Slate POM context, the support burden is not just about volume. It is about the specificity of questions. Partners, content operators, and internal stakeholders ask questions that span content ingestion, metadata standards, availability windows, billing, and regional licensing. The answers exist somewhere, usually spread across a combination of internal wikis, help center articles, and policy documents that no single person has fully memorized. Finding and synthesizing those answers in real time is exactly the kind of repetitive cognitive work that burns out POM teams and creates unnecessary bottlenecks.
A RAG-based chatbot felt like the right architecture. Retrieval-Augmented Generation grounds a language model's responses in documents you control, rather than relying on whatever a general-purpose model absorbed during training. The quality of the chatbot's answers would be directly tied to the quality of the content fed into it. That sounded clean and controllable. Then came the content problem.
User Question > RAG Retrieval > LLM Response > Grounded Answer > POM Escalation
The flow above is the goal. A user asks a question, the system retrieves the most relevant chunks from the knowledge base, feeds them as context to the language model, and returns a grounded answer. Only genuine edge cases escalate to a human. Getting there required solving the knowledge base problem first, and that turned out to be the hardest part of the whole project.
Trying to Scrape FAQ and Support Content
Building a RAG knowledge base starts with having content to put into it. The plan was to write a Python scraper that would pull down Prime Video's help center and FAQ pages, chunk the text, and use it as the foundation of the retrieval layer. In theory, this is a scripting problem. In practice, it became a genuinely frustrating exercise in content archaeology.
Prime Video's help center is not a flat collection of pages you can walk through in sequence. Content is deeply nested, conditionally rendered by account type and region, and frequently embedded inside pages in ways that do not follow predictable URL patterns. More critically, a large portion of the most useful FAQ content sits inside JavaScript-rendered components: expandable accordions, tabbed panels, and dynamic FAQ modules that do not exist in the raw HTML at page load. A standard HTTP-based scraper never sees them at all.
The Core Problem
Critical support content was embedded inside JavaScript-rendered accordion components and tabbed panels. Standard request-based scrapers returned empty shells of pages, missing the actual answers entirely. Even headless browser approaches required simulating user clicks to expose the content.
The approach that got the furthest used Playwright in Python, which spins up a real browser, waits for JavaScript to fully execute, then extracts the rendered content. Even then, some pages required simulating clicks on accordion toggles before the content became accessible. It worked, but it was slow, fragile, and sensitive to small page structure changes.
import asyncio
from playwright.async_api import async_playwright
from bs4 import BeautifulSoup
import json
# Target support pages — many returned partial or empty content
SUPPORT_URLS = [
"https://www.primevideo.com/help",
"https://www.primevideo.com/help/ref=atv_hp_nd_nav?nodeId=GTFM9LFBVV7DQPQJ",
"https://www.primevideo.com/help/ref=atv_hp_nd_nav?nodeId=GX7W4MSVDX6MHKRY",
]
async def scrape_page(page, url):
await page.goto(url, wait_until="networkidle")
await asyncio.sleep(2) # let JS fully settle
# Click any FAQ toggles / accordion headers to expand content
toggles = await page.query_selector_all(
"[data-testid='faq-toggle'], .a-expander-header, [aria-expanded='false']"
)
for toggle in toggles:
try:
await toggle.click()
await asyncio.sleep(0.3)
except:
pass # keep going if a toggle throws
html = await page.content()
soup = BeautifulSoup(html, "html.parser")
# Strip navigation, scripts, and decorative elements
for el in soup.select("nav, header, footer, script, style, [role='navigation']"):
el.decompose()
chunks = []
for block in soup.select("p, li, h2, h3, h4, [class*='answer'], [class*='content']"):
text = block.get_text(separator=" ", strip=True)
if len(text) > 50: # skip nav crumbs, button labels, short fragments
chunks.append({"text": text, "source": url})
return chunks
async def main():
all_chunks = []
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
for url in SUPPORT_URLS:
print(f"Scraping: {url}")
try:
chunks = await scrape_page(page, url)
all_chunks.extend(chunks)
print(f" Got {len(chunks)} chunks")
except Exception as e:
print(f" Failed: {e}")
await browser.close()
with open("faq_chunks.json", "w") as f:
json.dump(all_chunks, f, indent=2)
print(f"\nDone. Total chunks: {len(all_chunks)}")
asyncio.run(main())
Even with this setup, results were wildly inconsistent. Some pages returned rich, structured text. Others came back with marketing fragments, navigation breadcrumbs, and almost none of the actual support content. The Slate-specific documentation, the content that would have been most valuable for the chatbot, was either locked behind authenticated partner portals or embedded in internal tooling with no public-facing URL at all.
Design Around the Gap
After several iterations of the scraper, it became clear that waiting for a perfect knowledge base was going to block the project entirely. The practical decision was to build with what could be extracted, and be intentional and honest about how the chatbot handled everything else.
For content that scraped cleanly, it went through a chunking and embedding pipeline and became part of the retrieval layer. For content that could not be extracted, rather than having the chatbot guess or hallucinate a plausible answer, it was explicitly instructed to acknowledge the boundary of its knowledge and redirect the user to either a specific URL or a direct contact path for that topic.
This became one of the more interesting design decisions in the whole project. A chatbot that says "I do not have that information, but here is exactly where to find it" is genuinely more useful to a time-pressured POM partner than one that generates a confident-sounding answer it cannot verify. The redirect behavior had to be made explicit in the system prompt, with a curated list of fallback links and escalation contacts for the topics the knowledge base could not cover.
Rather than papering over gaps with LLM guesswork, the chatbot was designed to escalate gracefully. For any query it could not ground in retrieved content, it surfaced a direct link or a POM contact path. Honest boundaries are a feature, not a failure.
The workaround
SYSTEM_PROMPT = """
You are a support assistant for Prime Video's Slate POM workflow.
Answer questions using ONLY the retrieved context provided below.
Do not invent answers, speculate, or extrapolate beyond the context.
If the user's question is not covered by the context, do NOT attempt
to answer it. Instead, use one of these escalation responses:
For content ingestion or metadata issues:
"I don't have detailed documentation on that specific topic.
For Slate ingestion support, please visit the Video Direct
Help Center: https://videodirect.amazon.com/home/help
or reach out to your dedicated Slate POM directly."
For account or billing questions:
"For account and billing support, please visit:
https://www.primevideo.com/help or call 1-888-280-4331."
For urgent partner escalations:
"This sounds like it needs direct POM attention. Please
contact your Slate support lead or submit a ticket through
the Partner Portal."
Be clear and direct. Never apologize excessively. Always give
the user a concrete next step, even when you cannot answer.
"""
Writing the system prompt carefully was one of the highest-leverage decisions in the entire project. The way you instruct the model shapes its entire personality and reliability profile. A vague prompt produces a chatbot that sounds confident but cannot be trusted. A specific, bounded prompt produces one that users can actually rely on, even when its answer is "I do not know."
Don't Overengineer It
RAG architecture gets written about like it always requires serious infrastructure. It can. But for a proof of concept working from a limited knowledge base, the simplest implementation is usually the right starting point. The pipeline here was kept deliberately lightweight: clean and chunk the scraped content, generate text embeddings, store them in a local vector store, and at query time retrieve the most semantically relevant chunks to pass as context to the LLM.
For a project operating at this scale, with hundreds of documents rather than millions, a local vector store like ChromaDB handles retrieval comfortably without requiring any managed cloud infrastructure. There is no need to stand up a production vector database until the scale actually demands it.
The biggest practical learning from building the RAG pipeline was how much chunk quality matters upstream. Chunks that blended navigation text, breadcrumbs, and real content together produced retrieval noise. Spending time cleaning the scraped output before embedding it, stripping structural artifacts and short fragments, made a measurable difference in the relevance of what the retrieval layer returned at query time.
Using a Quick and Testable Chat UI Fast: Google Dialogflow
Once the RAG pipeline was working in Python, the next challenge was giving it a front end that did not require building a full custom chat interface from scratch. For a prototype that needs to be in front of stakeholders quickly, Google Dialogflow CX is one of the fastest paths from a working backend to a testable product.
Dialogflow provides a hosted conversational layer, a visual flow builder for designing conversation paths, and a one-line embed snippet that drops a functional chat widget onto any web page. The pattern used here was to treat Dialogflow as the UI and routing layer, and wire it to the custom RAG backend via a webhook. A user message comes into Dialogflow, gets forwarded to the Python RAG service, which retrieves context and calls the LLM, and sends the response back. Dialogflow handles the presentation. The RAG pipeline handles the intelligence.
Setting up Dialogflow's side of this is genuinely fast. You create an agent, configure a default fallback intent that routes all unmatched input to the webhook, and point the webhook URL at your RAG service. The most important configuration decision is making the default fallback the primary path rather than pre-defining intents for every possible question. Pre-defined intents are the old chatbot paradigm. Using an LLM means you want to send everything to the model and let the retrieval layer and system prompt do the work.
User Message > Dialogflow CX > Webhook to Rag > LLM Response > Chat UI
<script src="https://www.gstatic.com/dialogflow-console/fast/messenger/bootstrap.js?v=1">
</script>
<df-messenger
intent="WELCOME"
chat-title="Prime Video Slate Support"
agent-id="YOUR_DIALOGFLOW_AGENT_ID"
language-code="en"
></df-messenger>
<!-- Override widget CSS to match product brand -->
<style>
df-messenger {
--df-messenger-button-titlebar-color: #0F79AF;
--df-messenger-bot-message: #f6f5f2;
--df-messenger-user-message: #0F79AF;
--df-messenger-chat-background-color: #ffffff;
--df-messenger-send-icon: #0F79AF;
--df-messenger-font-family: 'Syne', sans-serif;
z-index: 9999;
}
</style>
Designing for What the Bot Does Not Know
One of the more counterintuitive design challenges was making the chatbot's knowledge boundaries feel like a feature rather than a failure. In a traditional rule-based FAQ bot, an unanswered question is a dead end. In an LLM-based system, there is a strong temptation to let the model generate something, because it almost always can produce something plausible-sounding. That instinct is exactly wrong for a high-stakes support context like Slate POM workflows.
Partners using this chatbot are often under real time pressure. A confidently wrong answer about a content ingestion deadline or a metadata field requirement does not just fail to help. It actively causes downstream problems. The design goal was to make the chatbot's honesty about its limits feel immediately recognizable, reassuring rather than frustrating, and above all actionable. When it does not know, it should say so clearly and tell you exactly where to go next.
There was also a visual design layer to this. The default Dialogflow Messenger widget styling is generic in a way that does not reinforce any product context. A few CSS variable overrides, tuning the color scheme, typography, and button styles toward the Prime Video visual language, made the experience feel considerably more considered, even at prototype fidelity. Branding a chatbot is not vanity. It signals to the user that this is an intentional product, not a hastily deployed script.
What This Project Actually Taught Me
The most important lesson from this project had nothing to do with RAG architecture or prompt engineering. It was about the fundamental dependency of any knowledge-retrieval system on the quality and accessibility of the knowledge itself. You can build the most sophisticated retrieval pipeline in the world, but if the content it is supposed to retrieve is locked behind JavaScript renders, authentication walls, and internal tooling with no scrapeable URL, the pipeline has nothing meaningful to work with.
For anyone building a similar system, the most valuable early investment is a content audit, not an architecture decision. Map what documentation exists, where it lives, what format it is in, and how accessible it is programmatically before writing a single line of scraping code. That audit will shape every downstream decision about what the chatbot can and cannot do.
The second big lesson was about scope management in AI prototyping. The temptation to keep expanding the knowledge base until the chatbot can answer everything is real, and it is a trap. A focused prototype that handles the most common POM questions reliably, and gracefully redirects on everything else, is more useful to stakeholders than a bloated system that attempts broad coverage and answers some of it badly. Scope confidence beats scope completeness at the prototype stage.
The project is not finished. The knowledge base gap is real, and filling it properly will require coordinating with the teams who own the internal documentation, not just scraping tools. But the architecture works, the fallback behavior is solid, and the prototype validated the core hypothesis: a well-designed RAG chatbot can meaningfully reduce POM dependency on the high-frequency, lower-complexity questions that currently take up a disproportionate share of team bandwidth.
Do a content audit before building anything. Design for what the bot does not know as carefully as for what it does. Use Dialogflow or a similar embed tool to get a testable UI in front of stakeholders fast. Treat graceful escalation, not coverage completeness, as the real measure of a support chatbot's quality. And write your system prompt like it is the most important product decision you will make, because it probably is.
