Penn State Great Valley | Real-Time Systems

Real-Time Distributed System
for Trend Analysis

A production-grade streaming pipeline that continuously ingests news articles from the Google News API via Kafka, routes data through Spark ETL into a polyglot storage layer (MongoDB for raw documents, Cassandra for time-series metrics), and applies NLP classification to surface trending topics, publisher frequency, and incident signals across Pennsylvania university coverage in near real-time.

●Live Kafka streaming pipeline ●4 Docker containers ●Polyglot storage: MongoDB + Cassandra ●NLP: Bag of Words + NLTK ●30+ publisher sources captured ●Sub-second ingest-to-storage latency

Apache Kafka Apache Spark MongoDB Cassandra Docker Python NLTK RapidAPI Google News API Power BI Hadoop / HDFS

Architecture

Pipeline & Methodology

Docker containers (Kafka, Cassandra, MongoDB, processing)

30+

Unique publisher sources captured in live collection session

101+

Articles archived in MongoDB in a single collection run

<1s

End-to-end ingest latency from API call to Cassandra write

Step 01

API Ingestion via RapidAPI

A Python KafkaProducer polls the Google News API via RapidAPI at regular intervals using keywords Pennsylvania universities and Penn State. Each article is serialized as UTF-8 JSON and published to the penn_news Kafka topic. A custom parse_timestamp() function normalizes all ISO, Unix-integer, and float timestamp formats before publishing to maintain time consistency in Cassandra.

Step 02

Kafka Broker and Topic Management

Apache Kafka runs as a Docker container at broker address 172.17.0.4:9092. The penn_news topic acts as a durable, partitioned message queue, decoupling producers from all downstream consumers. Kafka persists messages to disk, enabling any consumer (MongoDB writer, Spark processor) to resume from its last committed offset after a restart, guaranteeing zero message loss across the pipeline.

Step 03

MongoDB: Raw Document Archive

A dedicated Kafka consumer writes every article verbatim to MongoDB (database: news_archive, collection: articles, port 27017). MongoDB's schema-less document model accommodates structural variation across publishers with no migration overhead. A unique index on newsUrl enforces deduplication. Raw storage preserves original API payloads for future NLP reprocessing without re-querying the external API.

Step 04

Spark ETL and Cassandra Write

Apache Spark Structured Streaming consumes the penn_news topic in micro-batches, applies transformations (keyword extraction, entity tagging, source normalization), and writes structured output to Cassandra (keyspace: final_stream, table: realtime_news, port 9042). Cassandra's write-optimized, masterless architecture handles concurrent writes with low latency, with newsurl as the PRIMARY KEY for fast deduplication and point lookups.

Step 05

NLP Classification: Bag of Words

Collected articles are processed using NLTK with a Bag of Words model for two binary classification tasks: (1) Incident detection: identifying articles containing crisis or safety signals (7% of corpus). (2) Education classification: tagging articles as education-related (39%) versus general coverage. NLTK stop-word removal and frequency analysis produce the word cloud confirming domain concentration on Pennsylvania higher education.

Step 06

Reporting and Dashboard Delivery

Power BI connects to Cassandra for live dashboard refresh. Delivered visualizations include: Top Publishers by Article Count (30+ sources ranked), Word Cloud of trending topic terms from all 101 articles, Proportion of Incident-Related News (93% non-incident vs. 7% incident), and Education vs. General News classification (39% education, 61% general). P95 ingest-to-dashboard latency: under 10 seconds.

System Design

End-to-End Architecture

Real-Time Analytics Architecture: API to Dashboard

Docker Containers (Live Session)

Container	Image	Port(s)	CPU	Role
psu-cassandra-container	psu-cassandra	9042	0.62%	Real-time structured storage (final_stream.realtime_news)
psu-mongodb-container	psu-mongodb	27017	1.07%	Raw document archive (news_archive.articles)
blissful_volhard	psu-ubuntu	N/A	3.62%	Processing and ETL microservice environment
competent_mcnulty	psu-kafka	2181, 9092	100.58%	Message broker, actively streaming (high CPU is expected under load)

Data Layer

Polyglot Storage Schemas

Long-Term Storage

MongoDB: news_archive.articles

Field	Type	Note
_id	ObjectId	Auto-generated on insert
title	String	Article headline
newsUrl	String	Unique index, deduplication key
snippet	String	Lead paragraph from API
image	String	Cover image URL (nullable)
publisher	String	Source organization name
timestamp	Unix epoch (ms)	Normalized by parse_timestamp()

Schema-less JSON storage. No migrations needed as API structure evolves. 101+ documents collected.

Real-Time Storage

Cassandra: final_stream.realtime_news

Field	Type	Note
newsurl	text	PRIMARY KEY, deduplication
processed	boolean	False on insert, updated after NLP
publisher	text	Source organization
snippet	text	Article lead text
timestamp	timestamp	Parsed via datetime.fromisoformat()
title	text	Article headline

Write-optimized, LZ4-compressed. Replication factor 1. SizeTieredCompaction. Bloom filter FP rate 0.01.

Reporting System

NLP Analysis and Publisher Intelligence

Design Decisions and Outcomes

Kafka: Streaming Backbone

Offsets, partitions, and topic replication contributed to both fault tolerance and performance. The publish-subscribe model was essential: decoupling the API producer from Spark, MongoDB, and NLP consumers made the system modular and extensible. Kafka's persistence meant that if Cassandra restarted, the consumer simply resumed from its last offset with zero message loss.

Polyglot Persistence

MongoDB and Cassandra served complementary roles. Cassandra's write-optimized schema was ideal for storing real-time analytics metrics, while MongoDB's flexible document model preserved the original API payloads for future reprocessing. This validated the principle of picking the right database for the right access pattern rather than forcing one tool to serve both workloads.

Containerization with Docker

All four services ran as isolated Docker containers, enabling local simulation of a distributed cluster. Kafka's CPU usage hitting 100.58% under active streaming confirmed that horizontal scaling via additional brokers and partitions is the correct path for higher throughput, rather than vertical scaling of a single container node.

Design Constraints