Every large internet company eventually builds the same unglamorous piece of infrastructure: a system that takes in files of wildly different shapes, whether HTML, PDFs, images, or video, and turns them into something consistent enough for other systems to use. At Cloudflare, that system is called Riviera, and it has quietly underpinned products that need to parse, normalize, and extract meaning from content flowing across the network.
Riviera's original job was straightforward in concept and hard in practice: take raw bytes from wherever they entered Cloudflare's network and produce clean, structured output that downstream services could rely on. That meant handling malformed markup, stripping boilerplate, detecting language, and pulling out metadata, all at a scale where a rare edge case in one file format becomes a routine occurrence across billions of requests.
What's changed is who's asking for that output. A few years ago, the consumers of Riviera's pipeline were mostly analytics systems and security tooling looking for patterns. Today, a growing share of demand comes from AI features, retrieval systems, and agents that need the same raw content but in a very different shape: chunked passages with clean boundaries, embeddings-ready text, and metadata that reflects how a model will actually use the material rather than how a human browser would render it.
That shift sounds small, but it touches nearly every layer of the pipeline. A parser tuned to preserve visual structure for a screenshot service isn't necessarily tuned to preserve semantic structure for a retrieval-augmented generation system. Extracting "the article text" is a different task than extracting "the passages a model should retrieve when a user asks about pricing." The team behind Riviera has had to rethink chunking strategies, metadata schemas, and even how failures are handled, since a parsing error that used to just blank out a preview image now risks silently corrupting the context an AI system reasons over.
One recurring theme in this kind of migration is the tension between generality and specialization. A universal content pipeline is valuable precisely because every product doesn't need to reinvent parsing logic for PDFs or reimplement HTML boilerplate removal. But AI workloads tend to have sharper, more specific requirements than the median consumer, and bolting those requirements onto a general-purpose system risks turning it into an unmaintainable pile of special cases. The alternative, standing up separate AI-specific pipelines, duplicates effort and inevitably drifts out of sync with the main system over time.
The practical answer tends to be a layered design: a stable core that handles ingestion, format detection, and basic normalization, with pluggable extraction and chunking modules that different consumers can configure for their own needs. That lets an AI Search feature ask for semantically coherent chunks with embeddings-friendly boundaries, while a legacy analytics job keeps getting the flat text extraction it has always relied on, all from the same underlying ingestion path.
There's also a reliability dimension that becomes more important once AI systems are in the loop. Traditional content pipelines can tolerate occasional bad output because a human is usually the final consumer and can shrug off a garbled preview. AI pipelines feed model context directly, and a subtle parsing mistake, like a table rendered as unstructured text or a caption merged into a paragraph, can produce a confidently wrong answer with no obvious failure signal. That has pushed platforms like Riviera toward more aggressive validation and toward exposing confidence signals alongside extracted content, rather than treating extraction as a black box that either works or silently fails.
What this points to more broadly is that the plumbing behind AI products is rarely built from scratch. It tends to be existing infrastructure, originally designed for a different era of the internet, adapted piece by piece to meet a new kind of consumer. The unglamorous work of content parsing turns out to be just as central to whether an AI feature feels reliable as the model sitting on top of it.