NYC Mobility
Data Platform
An end-to-end data engineering platform built around real NYC Yellow Taxi trip data. Raw parquet lands in S3, an event-driven Lambda kicks off a PySpark job on Databricks, and the cleaned, enriched output is written as Delta Lake tables governed by Unity Catalog and served through an RDS warehouse. Every piece of infrastructure, VPC, S3, IAM, RDS, DynamoDB, Lambda, CloudWatch, CloudTrail, and the Unity Catalog governance objects, is defined in Terraform and deployed through a GitHub Actions pipeline authenticated to AWS via OIDC, with no long-lived credentials anywhere.
What Was Built
A real month of NYC TLC Yellow Taxi trip data flows through a PySpark ETL job into two curated Delta Lake tables, with the full AWS and Databricks infrastructure defined as code around it.
One Codebase, Three Runtimes
The transform logic lives in a single module with no Databricks-specific or Glue-specific APIs. The same tested code runs unchanged as a local job, an AWS Glue job, and a Databricks notebook job, only the I/O layer changes per environment.
Governed, Not Just Stored
Curated tables live in Unity Catalog, not a bare S3 path. Access is granted at the catalog and schema level to named roles, not by handing out bucket credentials, so every read is attributable to a principal rather than an anonymous access log line.
Split Ownership by Design
Terraform owns AWS resources and Unity Catalog governance objects, the platform team's concern. A Databricks Asset Bundle owns the job definition, cluster spec, and schedule, the data engineering team's concern. Neither manages the other's resources.
End-to-End Architecture
Four stages: an event-driven trigger on new raw data, a PySpark transform stage, a governed Delta Lake storage layer, and a warehouse/BI serving layer, with monitoring and audit logging wrapped around all of it.
Curated Delta Lake Tables
Two analytics-ready tables come out of the ETL job, plus a run log that records data-quality metrics on every execution for lineage and audit purposes.
SELECT, data engineers get SELECT,
MODIFY, and CREATE_TABLE. Every read is
attributable to a principal.
Terraform, Security, and CI/CD
Eight Terraform modules, each scoped to one concern, wired together from a single root configuration and validated end to end.
Monitoring & Alerting
Three CloudWatch alarms cover the failure modes that would otherwise go unnoticed in a solo-maintained pipeline, all routed through a single SNS topic. CloudTrail logs both control-plane API calls and object-level reads and writes on the two data buckets specifically, not just management events.
| Alarm | Trigger Condition | Why It Matters |
|---|---|---|
| Trigger Lambda errors | > 0 errors in a 5 min window | This Lambda is the only thing that starts the ETL job. If it fails silently, the whole pipeline just stops running with no other signal. |
| RDS free storage low | < 2 GB free | Warehouse tables grow with every ingested month of trip data; catches disk pressure before a write fails outright. |
| RDS high CPU | > 80% for 15 min | Flags a runaway query or unexpected serving load against the warehouse before it degrades for every consumer. |
AWS::S3::Object data events scoped to the raw and processed buckets, so every object read and write is in the audit log, not only who created or deleted the bucket itself. The trail is multi-region with log file validation enabled, and writes to a dedicated, encrypted, public-access-blocked log bucket.
Visualization Layer: Where Grafana and Prometheus Fit
CloudWatch alarms answer "is something broken." A visualization layer on top answers "how is it trending," which is where these two tools split cleanly by what they're each actually good at monitoring here.
Grafana on CloudWatch
Amazon Managed Grafana, pointed at CloudWatch as a data source, gives one dashboard across Lambda invocations and errors, RDS storage and CPU, and DynamoDB read and write capacity, the same metrics the alarms already watch, but as trend lines instead of a threshold crossed. No new instrumentation needed, CloudWatch is already collecting all of it.
Prometheus for Spark Internals
CloudWatch has no visibility inside a Spark job, it only sees whether the Databricks job succeeded or failed. Spark ships a native Prometheus metrics sink exposing executor memory, GC time, stage duration, and shuffle read/write. Remote-writing that into Amazon Managed Service for Prometheus and graphing it in Grafana is what would catch a job that's succeeding but getting slower every run, well before it crosses a duration alarm.
Key Design Decisions
Choices made about how the system is structured, and the reasoning behind them.
Environment-Agnostic Transform Layer
All PySpark logic lives in a module with no Databricks or Glue-specific imports. Three thin entrypoints (local script, Glue job shim, Databricks notebook) call the same functions with different I/O paths. One set of tests covers all three execution environments.
Governance as a First-Class Layer
Curated tables are registered in Unity Catalog with named-role grants rather than being left as a bare S3 path with shared bucket credentials. Access control and data location are treated as part of the system design, not an afterthought bolted on before a demo.
Terraform and Asset Bundles, Deliberately Separate
AWS infrastructure and Unity Catalog governance objects are Terraform's responsibility. The Databricks job definition, cluster spec, and schedule are the Asset Bundle's responsibility. This mirrors how a platform team and a data engineering team actually divide ownership in practice.
Data Quality as a Recorded Metric
Every pipeline run writes its row counts and drop percentage to a run log rather than just returning a success or failure status. A run that quietly drops most of its input is visible as an anomalous number in that log, not a silent success.