The Data Backbone of LLM Systems

Any LLM application has four dimensions you must carefully engineer: the code, data, models and prompts. Each dimension influences the other. That's why you must learn how to track and manage each. The trick is that every dimension has particularities requiring unique strategies and tooling. That's why directly applying SWE and DevOps principles that apply to the code does not work for others.

This presentation will dig into the data dimension and how it looks when building LLM applications. We will start by exploring a general framework that acts as the foundation. We will look into how the data flows, focusing on the data and features pipeline and how the data should be stored to be correctly shared, versioned, processed and analyzed for RAG, training and inference. Next, we will zoom into how the data is accessed during LLM fine-tuning and within the inference pipeline, which can be implemented as an RAG workflow or as something more complex, such as an agent.

To fully understand how the framework works (for building LLM applications), we will look at two concrete use cases, architecting the data layer of the following systems:

• An LLM Twin: Your Digital AI Replica (Use case used in our LLM Engineer's Handbook)
• A Second Brain AI Assistant (Use case used in our latest open-source course, freely available on Decoding ML)

We will present specific implementation details, tooling, and problems during these two use cases to fully understand an LLM system's data-related generalities and particularities.