This feature applies only to Aiven hosted Kafka. On-premises Kafka is deprecated, and creating new topics on-premises was disabled summer 2021. For on-premises Kafka, see on-premises Kafka documentation.
These variables are made available inside the pod.
||Comma-separated list of HOST:PORT pairs to Kafka brokers|
||URL to schema registry|
||Username to use with schema registry|
||Password to use with schema registry|
||Client certificate for connecting to the Kafka brokers, as string data|
||Client certificate for connecting to the Kafka brokers, as file|
||Client certificate key for connecting to the Kafka brokers, as string data|
||Client certificate key for connecting to the Kafka brokers, as file|
||Certificate authority used to validate the Kafka brokers, as string data|
||Certificate authority used to validate the Kafka brokers, as file|
||Password needed to use the keystore and truststore|
||PKCS#12 keystore for use with Java clients, as file|
||JKS truststore for use with Java clients, as file|
||A timestamp of when the secret was created|
What happens on deploy?¶
When you deploy an application that requests access to Kafka, Naiserator will create an
AivenApplication resource in the cluster.
AivenApplication has a name that matches the deployed application, and the name of the secret to generate.
Naiserator will request that a secret with this name used in the deployment.
AivenApplication resource is created or updated, Aivenator will create a new service user and generate credentials.
These credentials are then inserted into the requested secret and used in the deployment.
If there is a problem generating the secret, this might fail your deployment.
In this case, Aivenator will update the
status part of the resource, with further information about the problem.
Testing your application¶
If you need to test that you have configured your consumer application correctly, you can use one of the kafkarator-canary topics. All applications (from all teams) have read access to the canary topics. All the canary topics receives a message on a fixed interval, containing a RFC3339 formatted timestamp. The only difference between each, is where the producer is located. The available canary topics are:
Using Kafka Streams with internal topics¶
This feature is only available in GCP clusters.
In some configurations of kafka streams your application needs to create internal topics. To allow
your app to make internal topics, you need to set
true in your application
When you do this you must configure Kafka Streams by setting the property
application.id to the
value of the env var
KAFKA_STREAMS_APPLICATION_ID, which will be injected into your pod
Accessing topics from an application on legacy infrastructure¶
If you have an application on legacy infrastructure (outside NAIS clusters), you can still access topics with a few more manual steps.
The first step is to add your application to the topic ACLs, the same way as for applications in NAIS clusters (see the previous section). Use your team name, and a suitable name for the application, following NAIS naming conventions.
To create a credentials for your application, you need to manually create the
AivenApplication resource that would normally be created by Naiserator.
--- apiVersion: aiven.nais.io/v1 kind: AivenApplication metadata: name: legacyapplication namespace: myteam spec: kafka: pool: nav-dev secretName: unique-name protected: true
Since Aivenator automatically deletes secrets that are not in use by any pod, you need to set the
protected flag to
This ensures that the secret will not be deleted by any automated process.
AivenApplication resources has been created, Aivenator will create the secret, using the name specified.
kubectl, extract the secret and make the values available to your legacy application.
When you no longer have a need for the credentials created in this way, delete the
AivenApplication resource, and make sure the secret is also deleted.
If you migrate the application to NAIS, the first deploy to NAIS will overwrite the
When this happens, it is no longer
In this case, it is recommended that you manually delete the protected secret when it is no longer needed.
Application design guidelines¶
Authentication and authorization¶
The NAIS platform will generate new credentials when your applications is deployed. Kafka requires TLS client certificates for authentication. Make sure your Kafka and/or TLS library can do client certificate authentication, and that you can specify a custom CA certificate for server validation.
Readiness and liveness¶
Making proper use of liveness and readiness probes can help with many situations. If producing or consuming Kafka messages are a vital part of your application, you should consider failing one or both probes if you have trouble with Kafka connectivity. Depending on your application, failing liveness might be the proper course of action. This will make sure your application is restarted when it is experiencing problems, which might help.
In other cases, failing just the readiness probe will allow your application to continue running, attempting to move forward without being killed. Failing readiness will be most helpful during deployment, where the old instances will keep running until the new are ready. If the new instances are not able to connect to Kafka, keeping the old ones until the problem is resolved will allow your application to continue working.