95. Case Study: A Simple Analytics Application with Streaming

As data grows at an ever-faster pace, the need to process and analyze data in real time grows along with it. The traditional batch-processing approach, where data is collected and processed periodically, is often insufficient for modern business needs, especially for use cases such as transaction monitoring, fraud detection, or transportation-mode analytics. For this reason, streaming data has emerged as an effective solution.

In this article, I will share a case study on building a simple analytics application using streaming. We draw this case study from the real world but simplify it so that it is easy to understand and reproduce. We will cover the architectural design, sample code, a simulation, a results table, and a flow diagram using mermaid. By the end, readers should have a clear picture of how to implement an end-to-end streaming application.

Case Study: Streaming Analytics for E-commerce Transactions

Case Description:
An e-commerce company wants to monitor payment transactions in real time in order to analyze the average transaction value per city and to detect spikes in transaction volume. Transaction data flows continuously (as a stream) into the system.

Solution Architecture

For this simple case study, we choose the following stack:

• Streaming process: Apache Kafka (messaging) and Apache Spark Streaming (analytics)
• Result visualization: A console and a simple table

Here is the flow diagram:

flowchart LR
    User(Platform User) -->|Transaksi| API[API Order Service]
    API -->|Push data| Kafka[(Kafka Topic)]
    Kafka -->|Ingest| Spark[Spark Streaming Job]
    Spark -->|Hasil Analitik| Output[Console/Tabel Hasil]

flowchart LR
    User(Platform User) -->|Transaksi| API[API Order Service]
    API -->|Push data| Kafka[(Kafka Topic)]
    Kafka -->|Ingest| Spark[Spark Streaming Job]
    Spark -->|Hasil Analitik| Output[Console/Tabel Hasil]

Transaction Data Schema

Each incoming transaction has the following data structure (in JSON format):

1{
2  "order_id": "OID123456",
3  "created_at": "2024-06-20T10:14:38Z",
4  "user_id": "USER789",
5  "city": "Jakarta",
6  "amount": 250000.0
7}

The key fields in this case study are city and amount.

Simulating the Data Stream

Before we get into the analytics code, we need a simulation of transaction data to send to Kafka. For the purposes of this case study, we will use a simple Python script that pushes data to Kafka.

 1# transaksi_producer.py
 2import json
 3import random
 4import time
 5from datetime import datetime
 6from kafka import KafkaProducer
 7
 8cities = ['Jakarta', 'Bandung', 'Surabaya', 'Medan']
 9producer = KafkaProducer(bootstrap_servers=['localhost:9092'])
10
11def generate_transaksi():
12    return {
13        "order_id": f"OID{random.randint(100000, 999999)}",
14        "created_at": datetime.utcnow().isoformat() + 'Z',
15        "user_id": f"USER{random.randint(100, 999)}",
16        "city": random.choice(cities),
17        "amount": float(random.randint(50000, 500000))
18    }
19
20while True:
21    transaksi = generate_transaksi()
22    producer.send('transaksi_topic', json.dumps(transaksi).encode('utf-8'))
23    print("Kirim:", transaksi)
24    time.sleep(random.uniform(0.2, 1.0))  # simulate a data stream

Analytics Processing with Spark Streaming

We use Spark Structured Streaming (Python API: PySpark) to read data from Kafka and then compute:

1. The average transaction value per city per time window (for example, 1 minute)
2. Detection of transaction volume spikes per city (for example: volume rising more than 50% over the previous window)

Here is an example of the analytics code:

 1# streaming_analitik.py
 2from pyspark.sql import SparkSession
 3from pyspark.sql.functions import from_json, col, window, avg, count
 4from pyspark.sql.types import StructType, StringType, FloatType, TimestampType
 5
 6# Data schema
 7schema = StructType() \
 8    .add("order_id", StringType()) \
 9    .add("created_at", StringType()) \
10    .add("user_id", StringType()) \
11    .add("city", StringType()) \
12    .add("amount", FloatType())
13
14# Create Spark Session
15spark = SparkSession.builder \
16    .appName("TransaksiStreamingAnalitik") \
17    .getOrCreate()
18
19spark.sparkContext.setLogLevel("WARN")
20
21# Read the stream from Kafka
22raw_df = spark \
23    .readStream \
24    .format("kafka") \
25    .option("kafka.bootstrap.servers", "localhost:9092") \
26    .option("subscribe", "transaksi_topic") \
27    .load()
28
29# Parse JSON
30transaksi_df = raw_df.selectExpr("CAST(value AS STRING) as json_str") \
31    .select(from_json(col("json_str"), schema).alias("data")) \
32    .select("data.*") \
33    .withColumn("created_at", col("created_at").cast(TimestampType()))
34
35# Windowed average per city
36avg_df = transaksi_df \
37    .groupBy(
38        window(col("created_at"), "1 minute", "30 seconds"),
39        col("city")
40    ).agg(
41        avg("amount").alias("avg_amount"),
42        count("order_id").alias("volume_transaksi")
43    )
44
45query = avg_df.writeStream \
46    .outputMode("update") \
47    .format("console") \
48    .option("truncate", False) \
49    .start()
50
51query.awaitTermination()

Output Sample (Console Table)

Detecting Transaction Volume Spikes

To detect a volume spike, we need to store the previous window’s volume and compare against it. In a production implementation, this is typically done with Redis, Cassandra, or stateful streaming. For this simple case study, however, it is enough to add spike-checking logic on top of Spark Structured Streaming. Here is an illustration using a rolling 1-minute window:

 1from pyspark.sql.window import Window
 2from pyspark.sql.functions import lag, col, expr
 3
 4# Add the windowed dataframe
 5windowed_df = avg_df
 6
 7# Using lag (not directly supported in Structured Streaming; solution: write to a sink, then process in a batch/separate layer)
 8# Alternative: Write to a sink (for example, a database) to compare later
 9
10# Pseudo-code simulation:
11# volume_lama = read_previous_window(city)
12# lonjakan = volume_baru > 1.5 * volume_lama

In a real system, Spark streaming usually pushes results to storage (for example, Redis), and the alerting logic then reads from that storage to detect spikes.

Discussion: The Engineer’s Perspective

As an engineer, here are some important lessons from this real-time streaming case study:

• Single Source of Truth: Kafka as a central event log makes the system easier to scale and more fault-tolerant.
• Windows and State: Windowing over batch intervals is important for aggregation and trend/spike detection. However, stateful streaming requires additional storage and performance tuning.
• Handling Late Data: Dealing with out-of-order or late-arriving data is very important, especially when data comes from many sources or IoT devices.
• Alert System (Notifications): Volume-spike detection can be wired up to a webhook or email notification to raise alerts in a real system.

End-to-End Flow Diagram

sequenceDiagram
    participant User as User
    participant API as API Service
    participant Kafka as Kafka Broker
    participant Spark as Streaming Analytic
    participant DB as Storage/Console

    User->>API: Kirim Transaksi
    API->>Kafka: Publish Event
    Kafka->>Spark: Stream Data Transaction
    Spark->>DB: Tulis hasil agregat/window
    Spark->>Spark: Analisis Spike
    Spark-->>DB: Notifikasi/Alert Lonjakan

sequenceDiagram
    participant User as User
    participant API as API Service
    participant Kafka as Kafka Broker
    participant Spark as Streaming Analytic
    participant DB as Storage/Console

    User->>API: Kirim Transaksi
    API->>Kafka: Publish Event
    Kafka->>Spark: Stream Data Transaction
    Spark->>DB: Tulis hasil agregat/window
    Spark->>Spark: Analisis Spike
    Spark-->>DB: Notifikasi/Alert Lonjakan

Conclusion

Through this case study, we have seen how a simple streaming analytics pipeline is built, from ingesting data, to real-time processing and aggregation, all the way to simple anomaly detection. The approach above can be applied and extended to a wide range of needs, from financial monitoring applications to logistics demand forecasting and IoT.

Productionizing a streaming solution requires mature systems integration, solid monitoring, and handling for various edge cases such as duplicate data, late-arriving data, scaling, and more. The foundation, however, remains event-driven design and stateful streaming, as demonstrated above.

I hope this case study broadens your perspective and serves as a starter kit for engineers looking to step into the world of streaming analytics!

Feel free to explore the sample code and modify it to suit your own project’s needs. Happy coding!