Gwen Shapira will share design and architecture patterns that are used to modernize data engineering. We will see how Apache Kafka, microservices and event streams are used by modern engineering organizations to efficiently build data pipelines that are scalable, reliable and built to evolve.

We'll start the presentation with a discussion of how software engineering changed in the last 20 years - focusing on microservices, stream processing, cloud and the proliferation of data stores. These changes represent both a challenge and opportunity for data engineers. Then I'll share some practical suggestions on how to design future-proof ETL given those changed.