Adjusting librdkafka settings

Adjusting librdkafka settings

Some random example:

<kafka>
    <max_poll_interval_ms>60000</max_poll_interval_ms>
    <session_timeout_ms>60000</session_timeout_ms>
    <heartbeat_interval_ms>10000</heartbeat_interval_ms>
    <reconnect_backoff_ms>5000</reconnect_backoff_ms>
    <reconnect_backoff_max_ms>60000</reconnect_backoff_max_ms>
    <request_timeout_ms>20000</request_timeout_ms>
    <retry_backoff_ms>500</retry_backoff_ms>
    <message_max_bytes>20971520</message_max_bytes>
    <debug>all</debug><!-- only to get the errors -->
    <security_protocol>SSL</security_protocol>
    <ssl_ca_location>/etc/clickhouse-server/ssl/kafka-ca-qa.crt</ssl_ca_location>
    <ssl_certificate_location>/etc/clickhouse-server/ssl/client_clickhouse_client.pem</ssl_certificate_location>
    <ssl_key_location>/etc/clickhouse-server/ssl/client_clickhouse_client.key</ssl_key_location>
    <ssl_key_password>pass</ssl_key_password>
</kafka>

Authentication / connectivity

Amazon MSK

<yandex>
  <kafka>
    <security_protocol>sasl_ssl</security_protocol>
    <sasl_username>root</sasl_username>
    <sasl_password>toor</sasl_password>
  </kafka>
</yandex>

https://leftjoin.ru/all/clickhouse-as-a-consumer-to-amazon-msk/

Inline Kafka certs

To connect to some Kafka cloud services you may need to use certificates.

If needed they can be converted to pem format and inlined into ClickHouse config.

Example:

<kafka>
<ssl_key_pem><![CDATA[
  RSA Private-Key: (3072 bit, 2 primes)
    ....
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
]]></ssl_key_pem>
<ssl_certificate_pem><![CDATA[
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
]]></ssl_certificate_pem>
</kafka>

See also

https://help.aiven.io/en/articles/489572-getting-started-with-aiven-kafka

https://stackoverflow.com/questions/991758/how-to-get-pem-file-from-key-and-crt-files

Azure Event Hub

See https://github.com/ClickHouse/ClickHouse/issues/12609

Kerberos

  <!-- Kerberos-aware Kafka -->
  <kafka>
    <security_protocol>SASL_PLAINTEXT</security_protocol>
    <sasl_kerberos_keytab>/home/kafkauser/kafkauser.keytab</sasl_kerberos_keytab>
    <sasl_kerberos_principal>kafkauser/kafkahost@EXAMPLE.COM</sasl_kerberos_principal>
  </kafka>

confluent cloud

    <yandex>
        <kafka>
        <auto_offset_reset>smallest</auto_offset_reset>
        <security_protocol>SASL_SSL</security_protocol>
        <ssl_endpoint_identification_algorithm>https</ssl_endpoint_identification_algorithm>
        <sasl_mechanism>PLAIN</sasl_mechanism>
        <sasl_username>username</sasl_username>
        <sasl_password>password</sasl_password>
        <ssl_ca_location>probe</ssl_ca_location>
        <!--
          <ssl_ca_location>/path/to/cert.pem</ssl_ca_location>      
        -->
        </kafka>
    </yandex>

https://docs.confluent.io/cloud/current/client-apps/config-client.html

How to test connection settings

Use kafkacat utility - it internally uses same library to access Kafla as clickhouse itself and allows easily to test different settings.

kafkacat -b my_broker:9092 -C -o -10 -t my_topic \
   -X security.protocol=SASL_SSL  \
   -X sasl.mechanisms=PLAIN \
   -X sasl.username=uerName \
   -X sasl.password=Password

Last modified 2021.08.24 : Code format corrections. (6fe6a9e5)