Skip to content

architecture.md

Latest commit

 

History

History
278 lines (179 loc) · 15.1 KB

architecture.md

File metadata and controls

278 lines (179 loc) · 15.1 KB
Raw

Architecture Design: Distributed Document Search Service

1. Overview

This service is a Node.js REST API for multi-tenant document search. It uses Elasticsearch for full-text search, Redis for caching and rate limiting, and a small HTTP surface that is intentionally easy to review and demo.

The implementation is a production-aware prototype designed to show the shape of a system that could support millions of documents and sub-second search. It emphasizes tenant isolation, safe query handling, observability, and reviewer-friendly simplicity rather than broad feature coverage.

2. Assumptions and Prototype Scope

  • The local deployment is Docker Compose based.
  • Elasticsearch serves as both the search index and the prototype document store.
  • A production system would usually add a durable database for document metadata and lifecycle state.
  • A production system would usually move indexing to an asynchronous queue and worker.
  • Static bearer tokens are prototype authentication, not a real identity provider.
  • Distributed tracing is not implemented in the prototype.

3. High-Level Architecture

flowchart LR
  Client[Client / Reviewer] --> API[Node.js Express API]
  API --> ES[(Elasticsearch)]
  API --> Redis[(Redis)]
  API --> Logs[Structured JSON logs via pino]
  API --> Metrics[Prometheus metrics via /metrics]
Loading

The API is the control point for authentication, authorization, request validation, tenant isolation, rate limiting, and response shaping. Elasticsearch handles document storage and search execution. Redis reduces repeated work by caching search and document GET responses and by enforcing request-rate limits. Logging and metrics are exposed directly from the API process, which keeps local review simple.

4. Request Flow

Search

  1. The client calls GET /search with Authorization and X-Tenant-Id.
  2. The API authenticates the token and binds the request to a tenant.
  3. The query is validated and normalized before it reaches Elasticsearch.
  4. Redis is checked first for a cached search response.
  5. On a cache miss, the API executes a safe Elasticsearch bool plus multi_match query constrained to the tenant.
  6. The result is cached in Redis and returned to the caller.

Document Get

  1. The client calls GET /documents/:id.
  2. The API authenticates the caller and enforces role access.
  3. Redis is checked first for a cached document.
  4. On a cache miss, the API reads the document from Elasticsearch using the tenant and document id.
  5. The result is cached and returned.

Create and Delete

  1. Writers can create documents with POST /documents.
  2. The document is indexed in Elasticsearch with tenant state included in the stored record.
  3. Deletes are soft deletes so the system preserves lifecycle state rather than removing history immediately.
  4. Cache entries for the affected document are invalidated after delete.

5. Database and Storage Strategy

The prototype uses a deliberately small storage stack:

  • Elasticsearch for full-text search and prototype document storage
  • Redis for read caching and rate limiting
  • No PostgreSQL or separate durable database in the prototype

This is a prototype simplification, not a production recommendation. The goal is to demonstrate the core distributed search behaviors, tenant isolation, indexing, search, caching, soft delete, rate limiting, and observability without adding a full metadata database and asynchronous indexing pipeline to a short reviewer exercise.

Why Elasticsearch for Search

The primary workload is full-text document search with relevance ranking. That makes a Lucene-based search engine a better fit than a relational database as the primary search engine.

Elasticsearch is designed as a distributed search and analytics engine built on Apache Lucene, optimized for fast indexing, relevance-ranked search, near-real-time retrieval, and scalable querying across large datasets. It also supports production-oriented search patterns such as analyzers, scoring, filtering, index mappings, sharding, replicas, and vector or hybrid search capabilities. For this service, Elasticsearch directly supports the assignment’s requirements around searching many documents with low latency and relevance ranking.

PostgreSQL can support basic text search through full-text indexes, and it would be a strong choice for authoritative relational metadata. However, PostgreSQL is not the best primary engine for this prototype’s core search path because the system is intended to demonstrate distributed search-engine design: tenant-specific indexes, search-specific query DSL, relevance scoring, cacheable search results, and search-engine scalability patterns.

Why Elasticsearch Instead of OpenSearch

OpenSearch was a valid alternative. Both Elasticsearch and OpenSearch are distributed Lucene-based search engines and both can serve this use case. OpenSearch is Apache 2.0 licensed and has strong AWS integration through Amazon OpenSearch Service.

Elasticsearch was chosen for this prototype because:

  1. Mature search ecosystem and documentation. Elasticsearch has a long-established ecosystem around search, indexing, relevance tuning, operational patterns, and the broader Elastic Stack.
  2. Strong fit for relevance and future AI search extensions. Elasticsearch continues to invest in semantic search, hybrid retrieval, relevance tooling, and machine-learning/search features.
  3. Performance-oriented search roadmap. Benchmark claims vary by workload and source, so they should not be treated as universal. The practical takeaway is that both Elasticsearch and OpenSearch require workload-specific benchmarking before a production commitment.
  4. Reviewer familiarity. Elasticsearch is commonly recognized in distributed-search architecture discussions and is easy for reviewers to reason about in a prototype.

The trade-off is licensing and ecosystem preference. OpenSearch is often the better choice when a project requires a permissive Apache 2.0 license, strong AWS-native alignment, or avoids Elastic’s licensing model. Elasticsearch is a reasonable prototype choice when the goal is to demonstrate mature search-engine architecture, relevance ranking, and operational search patterns.

Elasticsearch as Prototype Document Store

In production, Elasticsearch should usually not be the only source of truth for document metadata and lifecycle state. A production design would normally use a durable transactional database such as PostgreSQL for authoritative document metadata, tenant ownership, status transitions, audit fields, and deletion lifecycle. Elasticsearch would then act as a derived search index populated through an asynchronous indexing pipeline.

For this prototype, Elasticsearch stores:

  • document title
  • document content
  • metadata
  • tenant ID
  • indexing status
  • created and updated timestamps
  • soft-delete state

This keeps the implementation small and reviewer-friendly while still demonstrating the key search-service concerns. The prototype uses state-based indexing with statuses such as INDEXED, INDEX_FAILED, and SOFT_DELETED, and all search queries filter to status = INDEXED.

The production evolution would be:

PostgreSQL / durable DB = source of truth
Queue / stream = indexing pipeline
Elasticsearch = derived search index
Redis = cache and rate-limit state

This production model gives stronger transactional guarantees, auditability, retry semantics, and disaster recovery. The prototype intentionally compresses those layers to keep the exercise focused.

Why Not PostgreSQL as the Main Prototype Store

PostgreSQL would be valuable in production, but it was intentionally omitted from the prototype for three reasons:

  1. Scope control. Adding PostgreSQL would require schema design, migrations, transactional lifecycle handling, dual-write or outbox logic, and reconciliation between PostgreSQL and Elasticsearch. That is important in production but heavy for a compact prototype.
  2. Search is the main assessment focus. The assignment is about distributed document search, not relational CRUD. Elasticsearch better demonstrates the core search architecture.
  3. Reviewer-friendly implementation. Fewer moving parts make it easier to run, inspect, and validate the prototype. Docker Compose only needs the API, Elasticsearch, and Redis.

The documentation should still be explicit: production should add PostgreSQL or another durable database as the authoritative system of record.

Cache Layer: Why Redis

Redis is used for two separate concerns:

  1. Caching hot read paths
  2. Rate limiting

Redis is a good fit because both workloads need very fast, short-lived, key-value access.

Why Cache Search APIs

GET /search is cached because search queries can be expensive compared with simple key-value reads. A search request may require query parsing, index lookup, scoring, filtering, pagination, and result serialization. In real systems, many tenants repeatedly issue the same or similar queries, especially for common terms, dashboards, support workflows, or recently viewed data.

The search cache key includes:

search:{tenantId}:{normalizedQuery}:{page}:{size}

This design provides:

  • Lower latency for repeated searches
  • Reduced Elasticsearch load
  • Tenant isolation because tenant ID is part of the key
  • Stable caching because whitespace-normalized equivalent queries share the same key
  • Bounded staleness through a short TTL

The prototype uses a short search cache TTL of 60 seconds. This avoids complex broad invalidation while keeping stale search results short-lived. After a document is soft-deleted, a previously cached search response may briefly contain old results until the search cache expires. That is an accepted prototype trade-off and should be documented.

Why Cache Document APIs

GET /documents/:id is cached because document retrieval is a common read path and individual documents can become hot. For example, a user interface may repeatedly fetch the same document details after a search result click, or multiple users in the same tenant may open the same document.

The document cache key is:

document:{tenantId}:{documentId}

This design provides:

  • Fast repeated document reads
  • Reduced Elasticsearch load
  • Tenant-safe cache separation
  • Precise invalidation on delete

Unlike search cache, document cache can be invalidated directly because the document ID is known. When DELETE /documents/:id soft-deletes a document, the service deletes the corresponding document cache entry immediately. This prevents stale document detail reads after deletion.

The document cache TTL is longer than the search TTL because a single document lookup is easier to invalidate precisely:

Search cache TTL: 60 seconds
Document cache TTL: 300 seconds

Rate Limiting Layer: Why Tenant and Document Limits

Redis is also used for fixed-window rate limiting with INCR and EXPIRE.

The prototype has two rate-limit scopes:

rate:tenant:{tenantId}
rate:document:{tenantId}:{documentId}

Both are needed because they protect different failure modes.

Tenant-Level Rate Limiting

Tenant-level rate limiting protects shared infrastructure from one tenant consuming disproportionate resources. Without tenant-level limits, a single tenant could overload the API, Redis, or Elasticsearch and degrade service for other tenants.

Tenant-level limits help enforce:

  • fairness across tenants
  • abuse protection
  • noisy-neighbor isolation
  • predictable shared-resource usage

The default prototype tenant limit is:

100 requests per minute per tenant

Document-Level Rate Limiting

Document-level rate limiting protects hot documents from accidental or abusive repeated access. A single document can become a hot spot even if the tenant’s total traffic is otherwise reasonable. For example, many users may repeatedly open the same document, a client may retry the same document request in a loop, or an attacker may target a specific document ID.

Document-level limits help enforce:

  • hot-document protection
  • reduced repeated Elasticsearch/cache pressure
  • finer-grained abuse control
  • protection for document-specific read and delete paths

The default prototype document limit is:

30 requests per minute per tenant/document

The document rate-limit key includes both tenant ID and document ID so that limits do not leak across tenants and one tenant’s hot document does not affect another tenant’s hot document.

Redis Failure Behavior

The prototype treats Redis as an optimization layer, not the source of truth. If Redis is unavailable:

  • cache reads and writes degrade gracefully
  • rate limiting fails open
  • the API can continue serving requests through Elasticsearch

This favors availability during Redis outages. In production, that trade-off should be revisited depending on abuse risk. A stricter production system may use Redis Cluster, local fallback counters, API gateway throttling, or a managed rate-limiting service.

6. Data Model and Isolation

Each request is scoped by tenant. The tenant id comes from the request context and is validated against the authenticated token. That design keeps cross-tenant access out of the normal code path instead of relying on post-query filtering after the fact.

The prototype stores document content, metadata, timestamps, and lifecycle status in Elasticsearch. Search and retrieval are tenant-scoped, which means the same document id is not treated as globally visible.

7. Observability

The service exposes two primary observability surfaces:

  • Structured JSON logs via pino.
  • Prometheus-formatted metrics on /metrics.

The logging and metrics design avoids sensitive payload leakage and avoids high-cardinality labels. This keeps the prototype useful for review without creating a noisy or fragile metrics setup.

8. Scalability Notes

The design is intentionally close to how a scalable document search service would be organized:

  • Stateless HTTP API instances can be scaled horizontally.
  • Elasticsearch handles the search workload.
  • Redis absorbs repeated reads and supports rate limiting.
  • Tenant isolation is enforced at the API boundary.

For larger production deployments, the biggest architectural change would be separating document writes from indexing and adding a durable source of truth for document metadata. That would improve write durability, recovery, and reindexing workflows.

9. Security and Safety

The prototype keeps search execution constrained and avoids exposing raw internal state in normal responses. Search input is normalized and bounded, protected routes require authorization, and tenant mismatches are rejected early.

Because the auth model is static-token based, it is suitable for a demo but not for production identity management. A production rollout would replace the tokens with an identity provider and a more complete authorization model.

10. Operational Summary

The current implementation is a pragmatic reviewer-facing system:

  • It is easy to start locally.
  • It demonstrates multi-tenant behavior clearly.
  • It shows caching, rate limiting, and metrics in a visible way.
  • It leaves room for the production changes that matter most without pretending those changes already exist.