Skip to content

Implement produce-path encryption in kafka-cache-client #1688

@jfallows

Description

@jfallows

Describe the desired outcome from the user's perspective
As a Zilla operator, I want plaintext messages produced by clients to be encrypted by the kafka-cache-client before being written to the cache, so that the cache exclusively stores encrypted messages and plaintext is never persisted.

Acceptance criteria

  • On produce, call encryption.resolveFields(schemaId) + encryption.encryptField(fieldId, ...) per field using the model's FieldEncryption strategy
  • Embedded mode (EmbeddedFieldEncryption):
    • Writes zilla:key, zilla:edek, zilla:edata headers always
    • Writes zilla:efields header for structured models (avro, json, protobuf); omits for bytes model
    • zilla:edata binary format: [block-count: 1 byte][offset: 4 bytes | length: 4 bytes] × block-count
    • zilla:efields binary format: [field-id: 2 bytes] × block-count
    • block-count=0 is a valid sentinel for an unencrypted message passing through — no decryption overhead incurred
  • Registry mode: catalog encodes key material in payload prefix; no zilla: headers written
  • Store fully encrypted message in cache — plaintext is never stored
  • Header names configurable under options.encryption.headers (defaults: zilla:key, zilla:edek, zilla:edata, zilla:efields)

Additional context

bindings:
  north-kafka-cache-client:
    type: kafka
    kind: cache_client
    options:
      encryption:
        headers:
          key: zilla:key       # default
          edek: zilla:edek     # default
          edata: zilla:edata   # default
          efields: zilla:efields  # default

zilla:efields is always written on produce for structured models even when no authorization policy is currently configured — consumer permissions may change after messages are stored.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions