Abstract:

In this talk, we will explore what data pipelines are and show how they fit into a modern data ecosystem.

Everyone is of the view that data is the newfound "Oil". In as much as this proposition may hold some water, what must not be lost on us is the fact that the value of the oil is in its potential after it has been refined and delivered to the consumer. Also like oil, it takes efficient pipelines to deliver data through each stage of its value chain.

We will discuss what Data Pipelines are, who builds Data Pipelines and why we build Data Pipelines. We would also explore concepts that have become industry standard and set the stage for best practices in implementing pipelines.

This talk is for Pythonistas who are found thinking of "what in the world is Data Engineering all about" or want to venture into the Data Engineering world.