From File Storage to AI Fuel
For years, the job of a content pipeline was straightforward: accept a file, store it, and serve it back when asked. Cloudflare's internal system for this work, referred to as Riviera, grew up in that world. But the rise of large language models has changed what "processing content" actually means, and Riviera has had to change with it.
The shift is visible in how Cloudflare's Workers AI platform now treats documents. Instead of just storing a PDF or an image, the pipeline increasingly needs to turn that file into something a model can reason over: clean text, described images, and structured metadata.
Turning Documents Into Model-Ready Text
A good example is Cloudflare's markdown conversion capability inside Workers AI. When a PDF comes in, the system doesn't just dump raw text. It first tries to read the document's internal structure, the tagged tree of headings, paragraphs, and lists that some PDFs already carry. If that structure exists, the pipeline uses it to produce a properly organized markdown document instead of a wall of disconnected text.
When a page contains images, the pipeline doesn't skip them either. It runs object detection first, then feeds the detected objects into a prompt for an image-to-text model, currently a Gemma-family model hosted on Workers AI, so that visual content gets folded into the same document as a written description. Metadata and embedded structured data, such as JSON-LD, get pulled out too, and can be appended or stripped depending on what the caller wants.
Why This Matters Beyond Cloudflare
This pattern, ingest anything, normalize it into clean text, describe what can't be read as text, is becoming the baseline expectation for any content platform that wants to sit in front of an AI system. A retrieval pipeline is only as good as the text it hands to a model, and a huge share of real-world content still arrives as scanned PDFs, screenshots, or slide decks rather than clean plain text.
For teams building on Workers AI, the practical upshot is that document ingestion stops being a separate, brittle preprocessing script and becomes part of the platform itself. That matters for anyone building retrieval-augmented systems, internal knowledge search, or agent tooling that has to work with whatever files a business actually has lying around, not the tidy text files a demo usually assumes.
As more of the internet's content gets consumed by agents rather than human eyes scrolling a page, pipelines like Riviera are quietly becoming as important as the models themselves. The model gets the credit, but it can only reason as well as the text it's given.