Skip to content

feat: add kafka connection#4036

Open
Yisaer wants to merge 5 commits intolf-edge:masterfrom
Yisaer:kfk_connector
Open

feat: add kafka connection#4036
Yisaer wants to merge 5 commits intolf-edge:masterfrom
Yisaer:kfk_connector

Conversation

@Yisaer
Copy link
Copy Markdown
Collaborator

@Yisaer Yisaer commented May 6, 2026

Summary

Add Kafka named connection support and allow Kafka sink configs to reuse broker connection settings through connectionSelector.

Changes

  • Added a stateless Kafka connection implementation.

    • Parses Kafka broker, TLS, and SASL settings.
    • Uses kafka-go dial as the health check.
    • Does not hold producer, consumer, or long-lived Kafka network resources.
    • Registers the kafka connection provider when the Kafka implementation package is loaded.
  • Updated Kafka sink metadata.

    • Added connectionSelector.
    • Marked Kafka broker, SASL, and TLS-related fields as connection-related.
  • Added test coverage.

    • Unit coverage for Kafka connection provisioning and registration.
    • Config merge coverage for Kafka sink connectionSelector.
    • FVT coverage for memory source -> Kafka sink using a Kafka named connection.
  • Added Kafka FVT as a standalone test job in the main test workflow.

    • Starts Redpanda locally.
    • Creates a Kafka named connection.
    • Creates a Kafka sink rule using only connectionSelector + topic.
    • Verifies the message is written to Kafka.

Verification

  • go test ./extensions/impl/kafka -run 'TestKafkaConnection'
  • go test ./internal/topo/node/conf -run TestOverwriteKafkaConnectionProps
  • go test -trimpath -tags='full deadlock' -count=1 -v ./fvt -run TestKafkaConnectionSelectorSinkE2E
  • go test -tags full ./internal/binder/io
  • python3 -m json.tool extensions/sinks/kafka/kafka.json

Note: The final workflow-only adjustment was pushed without rerunning local tests per maintainer request.

Signed-off-by: Song Gao <disxiaofei@163.com>
@Yisaer Yisaer changed the title current feat: add kafka connection May 6, 2026
Signed-off-by: Song Gao <disxiaofei@163.com>
Yisaer added 3 commits May 6, 2026 13:03
Signed-off-by: Song Gao <disxiaofei@163.com>
Signed-off-by: Song Gao <disxiaofei@163.com>
Signed-off-by: Song Gao <disxiaofei@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant