Episode 34 - Data Engineering with Joe Crobak, Foundations of Smart Software
Now with extra mappers and reducers!
I talk to Joe Crobak of the Data Engineering Weekly newsletter about data engineering, and it's importance in modern software projects. We cover:
- The definition of data engineering
- Hadoop, MapReduce, and Spark
- How data engineering has evolved over the last decade.
- Joe's experiences working in government vs. startups. The US Digital service, and the paperwork reduction act.
Joe Crobak is software developer that has worked on several data-powered consumer products at Foursquare and other startups in NYC. He worked for two years in the federal government as a founding member of the United States Digital Service team at the Department of Health and Human Services. Joe's Data Eng Weekly newsletter, which covers distributed systems and big data, reaches over 11,000 subscribers. He lives in Washington, DC with his wife and two dogs.
Episode 13 on Applied Artificial Intelligence, where we also looked at organizational hurdles to creating AI product