Rpk through proxy

[fmp88]

Hi,

I am running a self hosted Redpanda cluster on a Hashicorp Nomad Cluster with Consul for DNS.

The cluster works fine and the redpanda console can connect to the cluster and configure it.
Then I was trying to connect to the cluster from my local machine through Hashicorp Boundary Proxy accessing the Consul Mesh and this only works partially.

I can access the admin port for the cluster health, but for example creating a topic does not work and I receive a timeout

./rpk topic create foo -X brokers=127.0.0.1:9092 -X tls.enabled=true
unable to create topics [foo]: unable to dial: dial tcp 10.114.0.21:24490: i/o timeout

This is the listener part of my config. I tried using multiple listeners at some point, but this also didn't help

    rpc_server:
        address: 0.0.0.0
        port: {{ env "NOMAD_PORT_rpc" }}
    advertised_rpc_api:
        address: {{ env "attr.unique.network.ip-address" }}
        port: {{ env "NOMAD_HOST_PORT_rpc" }}
    rpc_server_tls:
        key_file: /secrets/tls/broker/key.pem
        cert_file: /secrets/tls/broker/cert.pem
        truststore_file: /local/tls/root.pem
        enabled: true
        require_client_auth: true
    kafka_api:
        - address: 0.0.0.0
          port: {{ env "NOMAD_PORT_kafka" }}
          name: mtls_listener 
    advertised_kafka_api:
        - name: mtls_listener
          address: {{ env "attr.unique.network.ip-address" }}
          port: {{ env "NOMAD_HOST_PORT_kafka" }}
        - name: proxy_mtls_listener
          address: 127.0.0.1
          port: {{ env "NOMAD_PORT_kafka" }}
    kafka_api_tls:
        - name: mtls_listener
          key_file: /secrets/tls/broker/key.pem
          cert_file: /secrets/tls/broker/cert.pem
          truststore_file: /local/tls/root.pem
          enabled: true
          require_client_auth: true
        - name: proxy_mtls_listener
          key_file: /secrets/tls/broker/key.pem
          cert_file: /secrets/tls/broker/cert.pem
          truststore_file: /local/tls/root.pem
          enabled: true
          require_client_auth: true
    admin:
        - address: 0.0.0.0
          port: {{ env "NOMAD_HOST_PORT_admin" }}
    admin_api_tls:
        key_file: /secrets/tls/broker/key.pem
        cert_file: /secrets/tls/broker/cert.pem
        truststore_file: /local/tls/root.pem
        enabled: true
        require_client_auth: true

Appreciate any help,

Thanks

[dotnwat]

@rockwotj did i see something recently in slack about proxy support in rpk (or may rpcn?)

[rockwotj]

That was maybe for HTTP in RPCN?

I think using the kafka API via a proxy is tough. There is this step in the protocol where the all the brokers addresses are to directly connect to in the metadata request. I think you'd need to either tell the broker to advertise a different address for this client, or you need to do something like muck with /etc/hosts to get this working.

Honestly it's probably best to SSH into the internal network and use RPK there.

[fmp88]

thank you @dotnwat and @rockwotj that answer is good for me.
I just wanted to be sure not to have missed something that would have allowed this scenario.

Will probably start some short lived service inside that network to do the configurations for now