Field Notes
Notes on data-powered systems, AI workflows, architecture, automation, and the practical work of turning messy problems into useful software.
2026-07-31
Agentic development is turning data engineers into software producers—building locally, proving value with users, and distributing through platforms the enterprise already trusts.
Read field note
2026-06-07
Dashboards are wine lists: accurate, organized, and silent about what any of it means. What stakeholders always wanted was the sommelier — the story that turns data into a decision. AI can finally play that role: it reads datasets it has never seen, finds meaning across sources, and writes the narrative. And once you have table service, the legacy BI tool starts looking like what it always was — a narrow web app. From level-1 metrics to AI storytelling, and from dashboards to full-stack data apps.
Read field note
2026-05-07
Most data work does not fail because one runner is too slow. It fails because the track between source systems, transformations, cloud platforms, models, reports, and business feedback is too long. Agents compress that distance by carrying context across the whole lifecycle—and turning a sequence of handoffs into one continuous loop.
Read field note
2026-04-22
The old data team moved information into one room. Agentic coding environments give the function a new control room—one that can orchestrate CRM, product usage, support, and relationship context into complete business workflows. The on-demand QBR shows what that role can become.
Read field note
2026-03-29
The warehouse gives agents a curated history of the business. Live systems reveal what is happening now and provide the controls required to change it. Operational agents need both: context from the museum, action in the city, and a governed path between them.
Read field note
2026-02-01
Git rarely felt worth the effort in solo, UI-driven data work. Agentic coding changed both sides of that calculation: agents make more changes than one person can track from memory, while also making version control almost effortless to operate. Git becomes the team’s memory, review surface, work ledger, and record of how the system evolved.
Read field note
2026-01-11
SaaS succeeded by abstracting away infrastructure when building and running software was hard. That trade-off made sense for a long time. But as infrastructure becomes cheaper and AI lowers the cost of building custom solutions, the economics that justified renting platforms are starting to break down. This post explores why that shift is putting pressure on traditional SaaS models—and what it means for teams deciding what they should actually own versus outsource.
Read field note
2025-12-21
As data engineering matured, factories became the default. Assembly lines brought reliability when tools were weak, storage was expensive, and mistakes were costly. But the constraints changed, and the architecture didn’t. Today, we still build stacks by habit—pre-aggregating, orchestrating, and industrializing work before we understand it. In this post, I argue for an anti-stack mindset: start with workshops, use strong tools at query time, delay pipelines until they’re earned, and treat factories as an optimization—not a prerequisite. The goal of modern data engineering isn’t to keep the line running. It’s to deliver insights.
Read field note
2025-11-30
As organizations grow, they transform from solo musicians into full orchestras — and that's when the real challenge begins. Data doesn't stop being collected; it stops being aligned. Systems drift, meanings diverge, and interpretation fragments. Data engineering exists to keep the orchestra playing the same piece, in the same key, at the same tempo. In this post, I explore why coordination becomes essential at scale, and why a coordinated data layer is now the foundation custom software gets built on — including an application built on top of a request board to get decisions made faster.
Read field note
2025-09-23
CSV got us started, but it struggles at scale: bloated files, messy schemas, and painfully slow queries. Parquet flips the model—storing data in compact, labeled columns that query faster, compress better, and form the foundation of modern lakehouses. In this post, I use a Lego analogy to show why Parquet isn’t just a file format upgrade, it’s the workshop behind today’s analytics.
Read field note
2025-08-31
Modern data architecture can feel like alphabet soup: databases, warehouses, lakes, lakehouses, catalogs. With Microsoft Fabric making Delta Lake the default, it’s never been more important to understand how these systems fit together. In this post, I use a kitchen metaphor to break down the strengths and weaknesses of each—lunchboxes, buffets, pantries, and chefs—before looking at DuckLake, a new approach that puts metadata where it belongs: in a database. The goal isn’t hype, it’s clarity—so you can design an architecture that feeds your business real insight.
Read field note
2025-06-09
Pandas is a fantastic tool for small datasets and quick analysis but hits limits when scaling or persisting state. DuckDB fills that gap by combining SQL-native querying, persistent local storage, and high performance—allowing data engineers to build scalable, reliable pipelines on their laptop without spinning up clusters. This post explores the practical differences between Pandas and DuckDB, real-world use cases, and why DuckDB is the smarter tool for modern data workflows.
Read field note
2024-12-23
Not every data request needs a fire alarm. This post cuts through the hype around “real-time” reporting—clarifying what it is, what it isn’t, and how to deliver fresh, actionable insights without burning down your team or your budget.
Read field note