Polymarket–Kalshi Arbitrage Bot

A real-time prediction-market ingestion, cross-exchange semantic matching, and web dashboard system. It streams Polymarket (Gamma + CLOB WebSocket) and Kalshi (REST + WebSocket) into Redis, runs a sentence-transformer pipeline to find likely equivalent markets, optionally verifies pairs with an LLM (Chutes.ai), and serves a FastAPI UI with live-style updates via Server-Sent Events (SSE).

Disclaimer: This software is for research and tooling only. It is not financial, legal, or trading advice. Trading involves risk; comply with all applicable laws and exchange terms.

---

Features

• Dual live feeds — Polymarket and Kalshi data normalized into Redis (polymarket:live, kalshi:live).
• Semantic matching — Hybrid scoring (embeddings, Jaccard, dates, entity hints) with incremental Redis-backed runs.
• Chunked scoring — Comparison step avoids multi-gigabyte temporary arrays on large candidate sets (important for full-market runs).
• Dashboard — Search, sort, pagination, profit filters, human feedback (Same / Different), SSE stream for table updates.
• Optional LLM verification — Twelve parallel Chutes workers can label pairs; UI still shows embedding matches when LLM is disabled or pending.
• Persistence — logs/cross_platform_matches.json plus Redis restore helpers for restarts.

---

Architecture

flowchart LR
  subgraph ingest [Ingestion]
    PM[Polymarket RT]
    KL[Kalshi RT]
  end
  subgraph store [State]
    R[(Redis)]
  end
  subgraph compute [Matching]
    CMP[Comparison pipeline]
    LLM[LLM verifiers optional]
  end
  subgraph ui [UI]
    WEB[FastAPI dashboard]
  end
  PM --> R
  KL --> R
  R --> CMP
  CMP --> R
  R --> LLM
  LLM --> R
  R --> WEB

Loading

Component	Entry	Role
Fetch	main.py → polymarket_rt, kalshi_rt	Discovery + WebSockets → Redis
Compare	src/comparison/main.py	Load PM/KL from Redis → match → comparison:live:*
LLM	src/comparison/llm_verifier_main.py	Fills llm field (Same / Different) via Chutes
Web	src/web/main.py	Uvicorn + static dashboard

Further detail on price refresh timing: docs/PRICE_SYNC_AUDIT.md.

---

Requirements

Dependency	Notes
Python	3.10+ recommended
Redis	Default redis://localhost:6379
Node.js + PM2	Used by ./run.sh for production-style multi-process runs
RAM	Full-market comparison + embeddings: plan for several GB (sentence-transformers + torch; GPU optional)
Network	Outbound HTTPS/WSS to Polymarket, Kalshi, and (optional) Chutes / Hugging Face

Install Python dependencies:

python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

---

Configuration

Copy and edit environment variables (see .env.example):

cp .env.example .env

Variable	Purpose
REDIS_URL	Redis connection URL
KALSHI_API_KEY / KALSHI_ACCESS_KEY	Kalshi API key ID
KALSHI_PRIVATE_KEY_PEM or KALSHI_PRIVATE_KEY_PATH	RSA private key for signed WS handshake
KALSHI_WS_URL	Override WebSocket URL (e.g. demo endpoint)
CHUTES_API_TOKEN	Enables 12× gradients-llm-* PM2 workers
HF_TOKEN	Optional Hugging Face token for model downloads
KAFKA_BOOTSTRAP / KAFKA_TOPIC	Optional Kafka sink (Polymarket path)

Polymarket public CLOB/Gamma usage typically needs no API keys. Kalshi live ticker WebSocket may return 401 without valid credentials; REST market discovery can still populate many markets.

---

Quick start (PM2)

From the repository root:

export REDIS_URL=redis://localhost:6379
export PYTHON="$(pwd)/.venv/bin/python3"
chmod +x run.sh
./run.sh

This will:

1. Optionally restore comparison:live from logs/cross_platform_matches.json if Redis is empty
2. Start fetch (Polymarket + Kalshi, live, all markets)
3. Wait ~50s for initial Redis population
4. Start 12 LLM workers only if CHUTES_API_TOKEN is set
5. Start comparison loop and web dashboard

Dashboard: http://localhost:9777 (binds 0.0.0.0).

The default dashboard port 9777 is set in ecosystem.config.cjs to reduce clashes with other services on 8765. To change it, edit WEB_PORT in that file or run the web module manually with WEB_PORT set (see below).

run.sh commands

Command	Effect
./run.sh	Full PM2 workflow (default)
./run.sh --stop	Stop gradients-related PM2 apps
./run.sh --fetch-only	Fetch only
./run.sh --web-only	Web dashboard only (PM2)
./run.sh --restore-force	Stop writers, clear selected Redis keys, restore from JSON
./run.sh -- --engine polymarket --live	Foreground main.py (see main.py --help)

Logs:

pm2 logs gradients-fetch
pm2 logs gradients-comparison
pm2 logs gradients-web

---

Manual run (without PM2)

Terminal 1 — Redis (if not already running):

redis-server

Terminal 2 — Ingestion:

export REDIS_URL=redis://localhost:6379
.venv/bin/python -u main.py --engine both --live --all-markets

Terminal 3 — Comparison loop:

export REDIS_URL=redis://localhost:6379
.venv/bin/python -u src/comparison/main.py --redis --loop --loop-interval 10

Terminal 4 — Dashboard:

export REDIS_URL=redis://localhost:6379
export WEB_HOST=0.0.0.0
export WEB_PORT=9777
.venv/bin/python -u src/web/main.py

---

HTTP API (dashboard backend)

Method	Path	Description
GET	/	Dashboard (index.html)
GET	/api/matches	Filtered, sorted, paginated matches (query params: q, sort_by, page, per_page, profit_min, profit_max, …)
GET	/api/matches/stream	SSE stream; pushes JSON when payload changes
PATCH	/api/matches/hf	Body: pm_market_id, kl_ticker, hf: Same | Different

Static assets are under /static/.

---

Project layout

├── main.py                 # Launcher: polymarket / kalshi / both
├── run.sh                  # PM2 workflow helper
├── ecosystem.config.cjs    # PM2 app definitions
├── requirements.txt
├── .env.example
├── docs/
│   └── PRICE_SYNC_AUDIT.md
├── logs/                   # JSON snapshots, PM2 logs (created at runtime)
└── src/
    ├── polymarket_rt/      # Gamma + CLOB WS → Redis
    ├── kalshi_rt/          # REST + WS → Redis
    ├── comparison/         # Pipeline, embedder, LLM verifier, Redis helpers
    └── web/                # FastAPI app, static dashboard

---

Matching pipeline (summary)

1. Load Polymarket and Kalshi market snapshots from Redis.
2. Normalize text and build a category + token pre-filter (Jaccard threshold).
3. Encode texts with sentence-transformers/all-MiniLM-L6-v2.
4. Semantic prescreen (cosine ≥ 0.30) in chunks to limit peak memory.
5. Hybrid score; keep top-K Kalshi candidates per Polymarket market; persist to comparison:live:matches.
6. Price refresh jobs sync live token prices into the comparison payload for the UI.

The dashboard applies additional display filters (e.g. non-Politics, volume, both sides priced, positive implied edge). Rows with llm: "Different" are hidden; empty llm is treated as pending / embedding-only and remains visible unless other filters remove it.

---

Troubleshooting

Symptom	Likely cause	What to try
Empty dashboard, Redis has no comparison:live:matches	Comparison OOM’d or never finished first full pass	Ensure enough RAM/swap; watch pm2 logs gradients-comparison; confirm step 6 completes
Kalshi WS 401	Missing or invalid API credentials	Set keys from .env.example; try demo KALSHI_WS_URL for testing
Dashboard 404 on /api/matches	Wrong port (another app on same port)	Use 9777 or the port in ecosystem.config.cjs / WEB_PORT
No rows though Redis has matches	UI filters (profit, politics, prices)	Widen profit filter; confirm Kalshi/Polymarket prices in Redis
Stale LLM labels after restart	Redis empty	Use ./run.sh restore path or restore_cli / cross_platform_matches.json

---

Contributing

Issues and pull requests are welcome. When changing matching or filter logic, consider updating docs/PRICE_SYNC_AUDIT.md if latency or data flow shifts.

---

Acknowledgements

Uses public APIs and open models (e.g. Sentence Transformers, Hugging Face). Third-party trademarks belong to their owners.