SPEC-00: AIS Real-Time Processing System.md

SPEC-00: AIS Real-Time Processing System

Document Metadata

• Version: 1.0.0
• Status: APPROVED
• Last Updated: 2024-01-15

---

1. System Purpose

Sistema de recepção, parsing e análise de mensagens AIS (Automatic Identification System) em tempo real, com suporte a múltiplos pods em Kubernetes e análise inteligente via LLM.

2. Technical Stack

Component	Technology	Version
Backend Framework	Laravel	11.x
PHP Version	PHP	8.2+
Database	PostgreSQL	15+
Cache/Queue	Redis	7.x
Redis Module	ReBloom	2.x
AIS Parser	pyais (Python)	2.x
LLM Provider	Minimax.io	API v1
WebSocket	Laravel Reverb	1.x
Queue Worker	Laravel Horizon	5.x

3. High-Level Architecture

┌──────────────────────────────────────────────────────────────────┐
│                         INGESTION LAYER                          │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐                 │
│  │  Pod A     │  │  Pod B     │  │  Pod C     │  (Redundancy)   │
│  │  Receiver  │  │  Receiver  │  │  Receiver  │                 │
│  └─────┬──────┘  └─────┬──────┘  └─────┬──────┘                 │
│        │               │               │                         │
│        └───────────────┼───────────────┘                         │
│                        ▼                                         │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              DEDUPLICATION LAYER (Redis)                 │    │
│  │  SET NX (hot) → Bloom Filter (warm) → DB UPSERT (cold)  │    │
│  └─────────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌──────────────────────────────────────────────────────────────────┐
│                       PROCESSING LAYER                           │
│  ┌────────────────┐    ┌────────────────┐    ┌───────────────┐  │
│  │ Python Parser  │───▶│ Message Buffer │───▶│ Batch Insert  │  │
│  │ (pyais pool)   │    │ (50 msgs/2s)   │    │ (PostgreSQL)  │  │
│  └────────────────┘    └────────────────┘    └───────────────┘  │
└──────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌──────────────────────────────────────────────────────────────────┐
│                        ANALYSIS LAYER                            │
│  ┌────────────────┐    ┌────────────────┐    ┌───────────────┐  │
│  │ Analysis       │───▶│ LLM Service    │───▶│ Event         │  │
│  │ Trigger (3min) │    │ (Minimax.io)   │    │ Generator     │  │
│  └────────────────┘    └────────────────┘    └───────────────┘  │
└──────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌──────────────────────────────────────────────────────────────────┐
│                       PRESENTATION LAYER                         │
│  ┌────────────────┐    ┌────────────────┐    ┌───────────────┐  │
│  │ REST API       │    │ WebSocket      │    │ Dashboard     │  │
│  │ (JSON)         │    │ (Reverb)       │    │ (Livewire)    │  │
│  └────────────────┘    └────────────────┘    └───────────────┘  │
└──────────────────────────────────────────────────────────────────┘

4. Key Design Decisions

4.1 Deduplication Strategy

• Problem: Multiple pods receive identical messages from same antenna
• Solution: 3-layer deduplication (Redis SET NX → Bloom Filter → PostgreSQL UPSERT)
• Guarantee: Zero duplicates in database

4.2 Message Hash Generation

• Input: Raw NMEA sentence (normalized, lowercase, trimmed)
• Algorithm: SHA256 truncated to 32 chars
• Critical: NO timestamp in hash (must be deterministic across pods)

4.3 Batch Processing

• Buffer Size: 50 messages OR 2 seconds (whichever first)
• Insert Method: Bulk INSERT with ON CONFLICT DO NOTHING

4.4 LLM Analysis Trigger

• Time-based: Every 3 minutes
• Batch-based: Every 100 messages (configurable)

5. Performance Requirements

Metric	Target	Notes
Daily Message Volume	400,000	~5 msgs/sec average
Peak Message Rate	50 msgs/sec	During high traffic
Dedup Check Latency	< 1ms	99th percentile
End-to-End Latency	< 500ms	Message receipt to DB
Duplicate Rate in DB	0%	Hard requirement

6. Project Structure

ais-system/
├── app/
│   ├── Console/Commands/
│   │   └── AISListenCommand.php
│   ├── Services/
│   │   └── AIS/
│   │       ├── DeduplicationService.php
│   │       ├── ParserService.php
│   │       ├── ReceiverService.php
│   │       ├── MessageBufferService.php
│   │       └── AnalysisService.php
│   │   └── AI/
│   │       └── MinimaxService.php
│   ├── Models/
│   │   ├── AISMessage.php
│   │   ├── Vessel.php
│   │   ├── AnalysisSummary.php
│   │   └── Antenna.php
│   ├── Events/
│   │   └── AIS/
│   │       ├── MessageReceived.php
│   │       └── AnalysisCompleted.php
│   ├── Jobs/
│   │   └── AIS/
│   │       ├── ProcessMessageBatch.php
│   │       └── RunAnalysis.php
│   └── Http/Controllers/API/
│       └── AIS/
│           ├── MessageController.php
│           ├── AntennaController.php
│           └── AnalysisController.php
├── python/
│   └── ais_parser.py
├── config/
│   └── ais.php
├── database/
│   └── migrations/
│       ├── create_ais_messages_table.php
│       ├── create_vessels_table.php
│       ├── create_analysis_summaries_table.php
│       └── create_antennas_table.php
└── docker/
    ├── Dockerfile
    └── docker-compose.yml

7. Dependencies Between Specs

SPEC-00 (This Document)
    │
    ├── SPEC-01: Database Schema
    │       └── Required by: All services
    │
    ├── SPEC-02: Deduplication Service
    │       └── Required by: SPEC-04 (Receiver)
    │
    ├── SPEC-03: Parser Service
    │       └── Required by: SPEC-04 (Receiver)
    │
    ├── SPEC-04: Receiver & Buffer
    │       └── Depends on: SPEC-02, SPEC-03
    │
    ├── SPEC-05: Analysis & LLM
    │       └── Depends on: SPEC-01
    │
    └── SPEC-06: API & Events
            └── Depends on: SPEC-01, SPEC-05

8. Implementation Order

1. Phase 1: SPEC-01 (Database) → SPEC-02 (Deduplication)
2. Phase 2: SPEC-03 (Parser) → SPEC-04 (Receiver)
3. Phase 3: SPEC-05 (Analysis)
4. Phase 4: SPEC-06 (API & Events)

---

Related Documents

• SPEC-01: Database Schema & Migrations
• SPEC-02: Deduplication Service
• SPEC-03: Parser Service (PHP-Python Bridge)
• SPEC-04: Receiver & Message Buffer
• SPEC-05: Analysis Engine & LLM Integration
• SPEC-06: API Endpoints & WebSocket Events