Overview Scale Architecture Data Layer Infrastructure Monitoring Design
AWS · Databricks · PySpark · Terraform

NYC Mobility
Data Platform

An end-to-end data engineering platform built around real NYC Yellow Taxi trip data. Raw parquet lands in S3, an event-driven Lambda kicks off a PySpark job on Databricks, and the cleaned, enriched output is written as Delta Lake tables governed by Unity Catalog and served through an RDS warehouse. Every piece of infrastructure, VPC, S3, IAM, RDS, DynamoDB, Lambda, CloudWatch, CloudTrail, and the Unity Catalog governance objects, is defined in Terraform and deployed through a GitHub Actions pipeline authenticated to AWS via OIDC, with no long-lived credentials anywhere.

2.96M real NYC TLC trip records processed Same transform code runs local, Glue, and Databricks 8 Terraform modules, validated end to end Unity Catalog governed Delta Lake tables OIDC-authenticated CI/CD, zero static AWS keys
Python PySpark Delta Lake Databricks AWS Glue Terraform S3 Lambda RDS DynamoDB GitHub Actions
Scale

What Was Built

A real month of NYC TLC Yellow Taxi trip data flows through a PySpark ETL job into two curated Delta Lake tables, with the full AWS and Databricks infrastructure defined as code around it.

2.96M
Raw NYC TLC Yellow Taxi trip records ingested, January 2024, public dataset, no authentication required
2.72M
Clean trip records after validation: trip distance, fare amount, duration, and passenger count bounds
68,902
Zone-and-hour aggregate rows in the curated analytics table, demand and revenue by pickup zone
8
Terraform modules (VPC, S3, IAM, RDS, DynamoDB, Lambda, monitoring, Unity Catalog governance)

One Codebase, Three Runtimes

The transform logic lives in a single module with no Databricks-specific or Glue-specific APIs. The same tested code runs unchanged as a local job, an AWS Glue job, and a Databricks notebook job, only the I/O layer changes per environment.

Governed, Not Just Stored

Curated tables live in Unity Catalog, not a bare S3 path. Access is granted at the catalog and schema level to named roles, not by handing out bucket credentials, so every read is attributable to a principal rather than an anonymous access log line.

Split Ownership by Design

Terraform owns AWS resources and Unity Catalog governance objects, the platform team's concern. A Databricks Asset Bundle owns the job definition, cluster spec, and schedule, the data engineering team's concern. Neither manages the other's resources.

System Design

End-to-End Architecture

Four stages: an event-driven trigger on new raw data, a PySpark transform stage, a governed Delta Lake storage layer, and a warehouse/BI serving layer, with monitoring and audit logging wrapped around all of it.

DATA SOURCE INGEST + TRIGGER TRANSFORM (DATABRICKS / GLUE) STORE + SERVE NYC TLC Trip Data Public parquet, no auth 2.96M rows, Jan 2024 INGEST + TRIGGER S3 raw bucket ObjectCreated event Lambda trigger_etl calls Databricks Jobs API DynamoDB processed_files, idempotency PYSPARK ETL clean_trips enrich_trips zone_hourly_aggregates data_quality_summary runs on Databricks job cluster or Glue DELTA / UNITY CATALOG trips_clean zone_hourly_metrics pipeline_run_log catalog + schema grants storage credential, external location RDS Postgres curated warehouse tables materialized from Delta served to BI / ad hoc SQL private subnet, no public route Monitor CloudWatch CloudTrail SNS alerts
NYC TLC Public Dataset
S3 Raw Bucket
+
Lambda Trigger
PySpark ETL (Databricks / Glue)
Delta Lake + Unity Catalog
RDS Warehouse
Terraform owns AWS and Unity Catalog governance. The VPC, S3 buckets, IAM roles, RDS instance, DynamoDB table, Lambda function, CloudWatch alarms, CloudTrail trail, and the Unity Catalog storage credential, external locations, catalog, schema, and grants are all defined as Terraform modules under a single root configuration, one KMS key encrypts everything at rest.
Databricks Asset Bundles own the job. The actual ETL job definition, cluster spec, and schedule live in a Databricks Asset Bundle, deployed independently of the Terraform-managed infrastructure. Platform-level access changes and pipeline-level logic changes ship on separate tracks without touching each other's resources.
Data Layer

Curated Delta Lake Tables

Two analytics-ready tables come out of the ETL job, plus a run log that records data-quality metrics on every execution for lineage and audit purposes.

Table 01
trips_clean
Row-level cleaned and enriched trip data. Physically impossible records (zero or negative distance, zero or negative fare, negative or implausible duration, missing passenger count) are dropped. Adds trip duration, pickup hour and day of week, computed speed, fare per mile, and a fare-anomaly flag.
Table 02
zone_hourly_metrics
Curated aggregate table: trip count, total revenue, average trip distance, average duration, average fare per mile, and anomaly count, grouped by pickup zone, date, and hour. 68,902 rows for the ingested month, the table analytics and BI consumers actually query.
Table 03
pipeline_run_log
Written on every pipeline execution: raw row count, clean row count, dropped row count and percentage, and run duration. Makes data-quality regressions visible as a metric over time rather than a silent success or failure signal.
Governance
Unity Catalog
Both tables sit inside a Unity Catalog catalog and schema, with grants issued to named roles rather than raw bucket credentials: analysts get SELECT, data engineers get SELECT, MODIFY, and CREATE_TABLE. Every read is attributable to a principal.
Explicit Schema
No Schema Drift by Accident
The ETL job selects a known, explicit column list from the raw input rather than passing through whatever columns happen to exist. An upstream schema change fails loudly with a missing-column error instead of silently propagating into the curated layer.
Reproducible
Rebuildable From Source
Raw data is never committed to the repository, a small script re-fetches the same public NYC TLC dataset on demand. Combined with pinned dependency versions and Delta Lake's transaction log, both curated tables can be regenerated from source at any time.
Infrastructure

Terraform, Security, and CI/CD

Eight Terraform modules, each scoped to one concern, wired together from a single root configuration and validated end to end.

VPC Private and public subnets, NAT gateway, S3 gateway endpoint, security groups scoped to the data-processing tier and RDS specifically.
S3 Raw and processed buckets, KMS encryption, versioning, public access fully blocked, access logging to a dedicated log bucket.
IAM Three purpose-built roles, Glue execution, Lambda trigger, and the Databricks cluster instance profile, each scoped to only the buckets and actions it needs.
RDS Postgres warehouse in a private subnet, KMS-encrypted storage, master password generated by Terraform and stored only in Secrets Manager.
DynamoDB Idempotency table keyed on S3 object key, guards against duplicate pipeline triggers from S3's at-least-once event delivery.
Lambda S3-triggered function that checks DynamoDB, then calls the Databricks Jobs API to start the ETL run.
Monitoring CloudWatch alarms on Lambda errors and RDS storage/CPU, a multi-region CloudTrail trail with S3 data-event logging for both buckets.
Databricks Storage credential, external locations, catalog, schema, and grants, managed via the official Databricks Terraform provider.
OIDC over static keys. CI/CD authenticates to AWS through GitHub's OIDC federation, a short-lived token exchanged for a role, not a stored access key and secret. No AWS credential of any kind lives in this repository or in GitHub secrets.
One KMS key, one place to rotate. A single customer-managed key with automatic rotation encrypts S3, RDS, DynamoDB, and CloudTrail logs. Key policy and rotation live in exactly one place instead of being scattered across services.
Observability

Monitoring & Alerting

Three CloudWatch alarms cover the failure modes that would otherwise go unnoticed in a solo-maintained pipeline, all routed through a single SNS topic. CloudTrail logs both control-plane API calls and object-level reads and writes on the two data buckets specifically, not just management events.

AlarmTrigger ConditionWhy It Matters
Trigger Lambda errors > 0 errors in a 5 min window This Lambda is the only thing that starts the ETL job. If it fails silently, the whole pipeline just stops running with no other signal.
RDS free storage low < 2 GB free Warehouse tables grow with every ingested month of trip data; catches disk pressure before a write fails outright.
RDS high CPU > 80% for 15 min Flags a runaway query or unexpected serving load against the warehouse before it degrades for every consumer.
CloudTrail logs data events, not just management events. The trail's event selector explicitly includes AWS::S3::Object data events scoped to the raw and processed buckets, so every object read and write is in the audit log, not only who created or deleted the bucket itself. The trail is multi-region with log file validation enabled, and writes to a dedicated, encrypted, public-access-blocked log bucket.
One SNS topic, three alarms. All three CloudWatch alarms fan out through a single SNS topic rather than each wiring its own notification path, so adding a fourth alarm later, or changing where alerts go, is a one-line change in one place instead of a change repeated across every alarm resource.

Visualization Layer: Where Grafana and Prometheus Fit

CloudWatch alarms answer "is something broken." A visualization layer on top answers "how is it trending," which is where these two tools split cleanly by what they're each actually good at monitoring here.

Grafana on CloudWatch

Amazon Managed Grafana, pointed at CloudWatch as a data source, gives one dashboard across Lambda invocations and errors, RDS storage and CPU, and DynamoDB read and write capacity, the same metrics the alarms already watch, but as trend lines instead of a threshold crossed. No new instrumentation needed, CloudWatch is already collecting all of it.

Prometheus for Spark Internals

CloudWatch has no visibility inside a Spark job, it only sees whether the Databricks job succeeded or failed. Spark ships a native Prometheus metrics sink exposing executor memory, GC time, stage duration, and shuffle read/write. Remote-writing that into Amazon Managed Service for Prometheus and graphing it in Grafana is what would catch a job that's succeeding but getting slower every run, well before it crosses a duration alarm.

System Design

Key Design Decisions

Choices made about how the system is structured, and the reasoning behind them.

Environment-Agnostic Transform Layer

All PySpark logic lives in a module with no Databricks or Glue-specific imports. Three thin entrypoints (local script, Glue job shim, Databricks notebook) call the same functions with different I/O paths. One set of tests covers all three execution environments.

Governance as a First-Class Layer

Curated tables are registered in Unity Catalog with named-role grants rather than being left as a bare S3 path with shared bucket credentials. Access control and data location are treated as part of the system design, not an afterthought bolted on before a demo.

Terraform and Asset Bundles, Deliberately Separate

AWS infrastructure and Unity Catalog governance objects are Terraform's responsibility. The Databricks job definition, cluster spec, and schedule are the Asset Bundle's responsibility. This mirrors how a platform team and a data engineering team actually divide ownership in practice.

Data Quality as a Recorded Metric

Every pipeline run writes its row counts and drop percentage to a run log rather than just returning a success or failure status. A run that quietly drops most of its input is visible as an anomalous number in that log, not a silent success.