Building Reliable Data-Intensive Applications in Python

Abstract

As applications grow, the hardest problems are rarely about writing business logic—they come from handling scale, throughput, expensive queries, large datasets, and systems that begin to slow down under pressure.

This talk explores the practical realities of building data-intensive applications in Python, focusing on the engineering decisions that help systems remain performant and maintainable as data volume grows.

The session covers topics such as:

• Efficient data access patterns and avoiding expensive queries
• Understanding database bottlenecks and indexing strategies
• Handling high-throughput workloads with batching and asynchronous processing
• Memory and CPU considerations when processing large datasets
• Pagination strategies and why OFFSET pagination breaks at scale
• Caching, aggregation trade-offs, and reducing unnecessary computation
• Observability techniques for identifying performance bottlenecks in production

Rather than focusing on theory, the talk draws from practical production lessons and real-world backend systems.

Attendees will leave with a clearer understanding of how to approach performance, scalability, and reliability when building data-intensive Python applications.