Summary
by Ananth Packkildurai:
This newsletter focuses on data engineering and sometimes includes machine learning topics.
OnAir Post: Data Engineering Weekly
News
Free ELT Course from Dagster
Dagster is excited to announce the launch of ETL with Dagster, a comprehensive seven-lesson course. This free course guides you through practical ETL implementation and architectural considerations, from single-file ingestion to full-scale database replication.
Simply sign up at Dagster University to get started. Once enrolled, you can track your progress and learn at your own pace.
Thomas Kejser: Iceberg, The Right Idea – The Wrong Spec
In the article “Iceberg, the right idea, wrong spec,” the author presents several excellent points about the Iceberg spec, highlighting the operational complexities it entails. Storing metadata in this manner makes it significantly larger than necessary, leading to fragmented and bloated metadata, and the space management problem is a pressing issue.
The article is probably the beginning of the debate, along with Ducklake, about the next gen iteration of Lakehouse
Uber: The Evolution of Uber’s Search Platform
Shopify: Introducing Roast – Structured AI Workflows Made Easy
Sem Sinchenko: Why Apache Spark is often considered as slow
Meta: Collective Wisdom of Models: Advanced Feature Importance Techniques at Meta
About
Web Links
Videos
Debunking Data Contracts with Ananth
February 15, 2023 (01:04:00)
By: DataHeroes
Ananth Packildurai, Founder of Schemata & Editor Data Engineering Weekly shares his perspective on the market opportunity with Data Contracts and why customers should look at a Contract first Data Strategy.
Data Contracts & Domain Ownership w/ Ananth Packkildurai
May 23, 2022 (01:05:00)
By: Joe Reis
Ananth Packkildurai joins the show to chat about data contracts, domain ownership, metadata management, and his new project Schemata. Ananth is a wealth of knowledge and experience in the data engineering space. He also publishes the Data Engineering Weekly newsletter.
