Market Streaming Infrastructure

Scope

This repository groups the main components used to deploy, run, and analyze a distributed real-time market-data pipeline. The system ingests live exchange updates, reconstructs order-book state, computes derived streams, and supports offline analysis of the resulting data.

The project is accompanied by a technical-note series documenting the design, operational behavior, and timing effects of the pipeline. See Technical Notes.

Components

Order Book Engine and Feature Extractor

• Order Book Engine and Feature Extractor
  Flink-based processing component that reconstructs order-book state and computes derived microstructure features.

Feed Handlers

• KuCoin Feed Handler
  Tokio-based service for ingesting KuCoin live market-data updates.

• Binance Feed Handler
  Tokio-based service for ingesting Binance live market-data updates.

Snapshot Middleware

• KuCoin Snapshot Middleware
  Middleware service for retrieving and serving KuCoin order-book snapshots used during stream initialization and recovery.

• Binance Snapshot Middleware
  Middleware service for retrieving and serving Binance order-book snapshots used during stream initialization and recovery.

Infrastructure

• GitOps
  Kubernetes and FluxCD manifests for the streaming, middleware, Kafka/Flink/Druid, and observability stack.

• Infrastructure as Code
  Terraform configuration for provisioning the underlying infrastructure.

• Project Generated Protobuf Code
  Generated Project Protobuf code packages for the market-data pipeline.

• Schema Deployment
  Schema Registry deployment and related configuration.

• Binance Generated SBE Code
  Generated Binance SBE code used for decoding exchange messages.

Offline Analysis

• Data Analysis Infrastructure
  GKE and Terraform infrastructure for the Apache Druid-based offline analysis environment.

• Data Queries
  Apache Druid queries used for post-processing, empirical analysis, and technical-note figures.

Archived Dataset

• Apache Druid Analytical Dataset
  Archived Druid storage state (about 25 GiB) produced by the live pipeline and processed during the empirical analyses of the technical notes.

Technical Notes

Technical Note #1: Design Trade-offs and Failure Modes in a Real-Time Market Data Pipeline [PDF]

This note analyzes the pipeline design and its behavior under saturation, recovery, and failure conditions.

Technical Note #2: On the Temporal Structure of Distributed Pipelines [PDF]

This note analyzes the temporal behavior of the pipeline, focusing on how processing delays, timing gaps, bursts, and compression affect downstream signals.