Panel: Modern Data Architectures

Disclaimer: This summary has been generated by AI. It is experimental, and feedback is welcomed. Please reach out to info@qconlondon.com with any comments or concerns.

In the panel titled "Modern Data Architectures," experts discussed various aspects of evolving data architecture landscapes. The panelists included professionals working across different domains in the data architecture field, sharing their experiences and insights on the challenges and opportunities they have encountered.

  • Panelist Introductions: The panelists introduced themselves, highlighting their careers in software and data engineering and their transitions from roles such as machine learning and batch processing to data streaming and data infrastructure roles.
  • Career Transitions: Panelists shared their experiences in transitioning from software engineering to data engineering, discussing the challenges and importance of clearly differentiating between roles such as data engineers, data analysts, and software engineers.
  • Challenges in Data Architecture: The discussion touched on common issues, such as the need for data platform teams to bridge gaps between management demands for AI and the operational needs of application teams. The panel emphasized the increasing importance of data in application development and AI integrations.
  • Data Centric Architectures: There was a consensus on the growing relevancy of data-centric architectures, where data becomes a central component augmenting application processes rather than being a mere afterthought.
  • AI and Data Streaming: With the rise of AI applications, there is a pressing need for low-latency data streaming solutions. The panelists highlighted the importance of evolving data platforms to meet such requirements and addressed the significance of smart indexing and caching mechanisms.
  • Future Trends and Adaptations: The conversation explored future job roles and the mutual incorporation of data engineering and software engineering skills. The integration of modern data architectures with software engineering practices was seen as essential in staying current and effective in the industry.

Throughout the panel, the focus remained on leveraging modern data architectures to foster innovation while balancing technical and business needs.

This is the end of the AI-generated content.



Date

Wednesday Apr 9 / 01:35PM BST ( 50 minutes )

Location

Fleming (3rd Fl.)

Slides

Slides are not available

Share

From the same track

Session Data Architecture

Reliable Data Flows and Scalable Platforms: Tackling Key Data Challenges

Wednesday Apr 9 / 10:35AM BST

There are a few common and mostly well-known challenges when architecting for data. For example, many data teams struggle to move data in a stable and reliable way from operational systems to analytics systems.

Speaker image - Matthias Niehoff

Matthias Niehoff

Head of Data and Data Architecture @codecentric AG, iSAQB Certified Professional for Software Architecture

Session AI/ML

Achieving Precision in AI: Retrieving the Right Data Using AI Agents

Wednesday Apr 9 / 11:45AM BST

In the race to harness the power of generative AI, organizations are discovering a hidden challenge: precision.

Speaker image - Adi Polak

Adi Polak

Director, Advocacy and Developer Experience Engineering @Confluent, Author of "Scaling Machine Learning with Spark" and "High Performance Spark 2nd Edition"

Session Data Architecture

Beyond the Warehouse: Why BigQuery Alone Won’t Solve Your Data Problems

Wednesday Apr 9 / 03:55PM BST

Many organizations mistake the adoption of a data warehouse, like BigQuery, as the golden ticket to solving all their data challenges. But without a robust data strategy and architecture, you’re simply shifting chaos into the cloud.

Speaker image - Sarah Usher

Sarah Usher

Data & Backend Engineer, Community Director, Mentor

Session AI/ML

The Data Backbone of LLM Systems

Wednesday Apr 9 / 02:45PM BST

Any LLM application has four dimensions you must carefully engineer: the code, data, models and prompts. Each dimension influences the other. That's why you must learn how to track and manage each. The trick is that every dimension has particularities requiring unique strategies and tooling.

Speaker image - Paul Iusztin

Paul Iusztin

Senior ML/AI Engineer, MLOps, Founder @Decoding ML