Note: This video is 176 minutes long. Free users can process up to ~142 minutes. Sign up for full video processing.
📋 Summary
In this video, we build an end-to-end data engineering project using Databricks for an FMCG domain scenario that involves merging data from two companies: Atlon and Sports Bar. The technical stack includes Python, SQL, S3, Medallion architecture, Databricks dashboard, and Genie. The project addresses data discrepancies and chaos resulting from Atlon's acquisition of Sports Bar, requiring the creation of a unified and reliable data layer. We follow the Medallion architecture (bronze, silver, gold layers) to clean, transform, and merge data, ultimately creating a consolidated dashboard for both companies. The project showcases how to handle incremental updates and maintain a scalable, long-term solution using Databricks.