Skip to main content

Kafka topic

Overview​

Apache Kafkais an end-to-end event streaming platform that:

  • Publishes (writes) and subscribes to (reads) streams of events from sources like databases, cloud services, and software applications.
  • Stores these events durably and reliably for as long as you want.
  • Processes and reacts to the event streams in real-time and retrospectively.

Those events are organized and stored in topics. These topics are then partitioned over buckets located on different Kafka brokers.

Event streaming thus ensures a continuous flow and interpretation of data so that the right information is at the right place, at the right time for your key use cases.

tip

The Kafka Topic destination supports batch and real-time syncs.

Destination tab​

The following table outlines the mandatory and optional parameters you will find on the Destination tab.

Image 1: Define your Destination

The following parameters will help to define your data sync destination and how it functions.

ParameterDescriptionExample
DestinationMandatory. Select your destination from the drop down menu.Kafka Topic
Bootstrap ServersMandatory. Bootstrap Servers are a list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This parameter should a CSV list of "broker host" or "host:port"localhost:9092,another.host:9092
Topic NameMandatory. The name of the Kafka Topic that messages will be produced to.
Use SSLCheck this if you want to connect to Kafka over SSL
SASL MechanismMandatory. Select the SASL (Simple Authentication and Security Layer) Mechanism to use for authentication:
- None
- PLAIN
- SCRAM-SHA-256
- SCRAM-SHA-512
- OAUTHBEARER (default)
- OAUTHBEARER (oidc)

Use None with Use SSL for a connection whose only credential is a client certificate. See Encryption and authentication.
Test ConnectionYou can use the "Test Connection" button to ensure that your credentials are properly configured to access your destination. If configured correctly, a "Connection Successful" pop-up will appear. If configured incorrectly, a "Connection Failed" pop-up will appear along with a link to the applicable error logs to help you troubleshoot.

Encryption and authentication​

Every certificate and key field is stored encrypted by the platform, and every one of them accepts either PEM content pasted into the field or a reference to a secret (see Certificates and keys from a secret). The certificate fields under Provider Compatibility, for the key hint and for a token endpoint behind mutual TLS, also accept a path to a PEM file on the Connections host. The Encryption (TLS) fields and Private Key (PEM) accept content only: a file path there is read as though it were the PEM itself, and the connection fails.

Encryption (TLS)

ParameterDescription
CA Certificate (PEM)The CA chain that signs the broker's certificate. Needed when that certificate is not signed by a public CA.
Client Certificate (PEM)The certificate Cinchy presents to the broker. Required for mutual TLS, which is a certificate-only connection with SASL Mechanism set to None.
Client Key (PEM)The private key for that client certificate. Encrypted. The destination has no passphrase setting for this key, so supply it unencrypted (a BEGIN PRIVATE KEY or BEGIN RSA PRIVATE KEY block, not BEGIN ENCRYPTED PRIVATE KEY).

Authentication, for OAUTHBEARER (oidc)

ParameterDescription
Token Endpoint URLMandatory. Where Cinchy requests an access token. Must be HTTPS, except on a loopback address.
Client IdMandatory. The client ID registered with your identity provider.
Client AuthenticationMandatory. How Cinchy proves its identity to the token endpoint: Client Secret sends a shared secret; Private Key (JWT) signs an RFC 7523 section 2.2 client assertion (private_key_jwt), which is what providers mean when they say they issue no client secrets; JWT Bearer Assertion sends a signed assertion as the authorization grant itself (section 2.1), which few providers accept.
Client SecretRequired when Client Authentication is Client Secret. Encrypted.
Private Key (PEM)Required when Client Authentication is Private Key (JWT) or JWT Bearer Assertion. The key that signs the assertion. Encrypted.
Signing AlgorithmDefaults to RS256. The RS, PS and ES families are available under Private Key (JWT), where Cinchy signs the assertion; financial-grade providers reject RS256 and require PS256 or ES256. Under JWT Bearer Assertion the choice is RS256 or ES256.
ScopeThe scope requested from the token endpoint, if your provider requires one.
ExtensionsSASL extensions sent to the broker with the token, as comma-separated key=value pairs.
Assertion ClaimsThe assertion's iss, sub, aud, expiry, not-before and whether to include jti. The defaults are what OIDC requires, with iss and sub set to the client ID and aud to the token endpoint, so override them only where your provider documents something else.
Provider CompatibilitySettings only some providers ask for: the key hint that identifies your registered public key (kid, or a certificate thumbprint in the x5t or x5t#S256 header), extra assertion claims, extra token-request parameters and headers, and a client certificate for a token endpoint that is itself behind mutual TLS. A standards-conformant provider needs none of them.
Which key hint to use

A provider holding more than one registered public key needs a hint to know which one verifies the assertion, and many providers start requiring one at your first key rotation rather than at registration. Use the header your provider's own documentation names. Setting a kid a provider cannot resolve fails outright, so leave it blank unless the provider publishes that key ID.

Certificates and keys from a secret​

Rather than pasting a certificate or key into the connection, you can keep it in the Cinchy Secrets table, including a secret whose value lives in AWS Secrets Manager or Azure Key Vault through the External Secrets Manager. This works for every field in the tables above, including Client Certificate (PEM), Client Key (PEM) and CA Certificate (PEM).

  1. Store the certificate or key in the Secrets table, one secret per value. For an external secret, the value in AWS or Azure must be in one of the formats the platform can read.
  2. In the Connections UI, open Info > Variables, add a variable of type Secret, and pick the secret. Add one variable per secret, for example clientCertificate and clientPrivateKey.
  3. In the destination field, enter the variable name with an @ prefix and nothing else, for example @clientCertificate in Client Certificate (PEM) and @clientPrivateKey in Client Key (PEM).
  4. Select Test Connection. The test resolves the variables before it connects, so a formatting problem in the stored secret shows up here rather than in the first job.

The value the secret resolves to must be exactly what you would otherwise paste into the field: the full PEM block, -----BEGIN ...----- and -----END ...----- lines included, with a real line break after the header line, between the base64 lines and before the footer line. A value whose line breaks were turned into spaces, or into the two characters \n, is rejected by the Kafka client as not being in PEM format.

Variable names

A variable reference is replaced wherever its name appears in the field, including as the start of a longer name. Give each variable a name that is not the beginning of another one: with variables named clientKey and clientKeyPassphrase, the reference @clientKeyPassphrase is partly replaced by the value of clientKey.

Rotating a certificate​

Because the connection reads the secret rather than holding a copy, rotating a certificate means updating the secret, in the Secrets table or in AWS or Azure. You do not edit the connection. When the new value takes effect depends on how the sync runs:

  • Batch sync: each job reads the secret when it starts, so the next job uses the new value.
  • Real-time sync: the Connections worker reads the secret when it starts processing the sync and keeps the Kafka connection it opened with it. The new value takes effect after the worker restarts or the data sync configuration is saved again.

Update the secret while the old certificate is still valid, so the broker accepts the connection throughout the change.

Next steps​

Appendix A​

Configuring a Dynamic Topic​

Cinchy v5.10 added the ability to use a '@COLUMN' custom formula to enable a dynamic parameterized Kafka Topic when syncing into a Kafka destination.

To use this functionality, follow the below instructions.

  1. Define which table and column you want to use for your dynamic topic. In this example, the table is [Product].[Tasks] and the column is "Quarter".
  2. Create a data sync using a Cinchy Table source and a Kafka Topic destination.
    1. Ensure that the column defined in step 1 is loaded into your Schema. Dynamic Kafka Topic
  3. In the 'Topic' field of the Kafka destination, insert "@COLUMN('<column-name>')". In this example, the formula would be @COLUMN('Quarter').

Dynamic Kafka Topic