How to Build a Database Without a Server

Presentation Title: How to Build a Database Without a Server

Speaker: Alex Seaton, Engineer at ArcticDB, Man Group.

Overview:

The presentation discusses the transition from managing traditional databases like MongoDB to a serverless model using ArcticDB. The focus is on leveraging client-side libraries and object storage to simplify the scalability and management of large data systems.

Main Topics Covered:

• Database Complexity: Managing large database installations can be costly and complex.
• Serverless Architecture: Discusses a model where a database can function using only a client-side library and object storage.
• ArcticDB: A solution developed by Man Group that interacts directly with object storage, reducing the need for traditional server-based databases.
• Data Structures and ACID Compliance: Examines the choice of core data structures and methods for ensuring ACID properties.
• Global State Management: Utilizes Conflict-free Replicated Data Types (CRDTs) to manage global state without extensive synchronization.
• Object Storage Evolution: Covers how object storage technology has advanced and its implications for database systems.

Challenges and Solutions:

• Discusses handling high latency in commodity storage and maintaining high read and write speeds.
• Emphasizes the importance of version control and data chunking for efficient data management.
• Explores potential future developments and improvements in object storage and serverless database design.

Conclusion:

The approach to databases presented offers a simplified, scalable solution that is particularly beneficial for distributed computing and large-scale data analytics, leveraging advanced features of modern object storage systems.

This is the end of the AI-generated content.

Modern data analytics workflows rely on scaling out to huge numbers of users and compute nodes. Managing database installations to handle this scale can be unsustainably complex and expensive. Is it instead possible to get rid of all this complexity and build a database with just a client-side library and object storage?

At Man Group we have evolved over time from managing one of the largest MongoDB installations in Europe to a serverless model where users interact directly with object storage using ArcticDB, our own database engine. What we've learnt from this should be interesting to anyone interested in distributed computing, not just database development.

We will focus on topics such as:

• Our choices around ACID and our core data structures
• How to manage global state with lock-free techniques such as CRDTs
• How we manage to work with relatively high latency commodity object storage
• How object storage has evolved over time, and how advanced it is becoming