I found some settings on NixOS Search
I’ve added these to my configuration.nix:
environment.systemPackages = with pkgs; [ apacheKafka ];
services.apache-kafka = {
enable = true;
settings = {
"broker.id" = 0;
"log.dirs" = [ "/tmp/kafka_logs" ];
listeners = ["PLAINTEXT://:9092"];
};
};
Installing “apacheKafka” updates my system with kafka-server-start.sh, and if I imperatively execute
kafka-server-start.sh server.properties
then I can create topics and pub/sub with the consumer and producer. Nmap says the port on 9092 is “open”
However, the server.properties is supposed to be generated by
services.apache-kafka = {
enable = true;
settings = {
"broker.id" = 0;
"log.dirs" = [ "/tmp/kafka_logs" ];
listeners = ["PLAINTEXT://:9092"];
};
};
If I use nmap, I can see the service on 9092 is closed.
How do I get NixOS to startup the apache kafka server automatically with the server.properties generated by the declaration in the configuration.nix?
Thanks for any help
I feel you, I had to look at a lot of Kafka docs to have a working setup. Here’s the config I use for a “kraft mode” Kafka, that is, a Kafka which doesn’t need Zookeeper:
services.apache-kafka = {
enable = true;
# Replace with a randomly generated uuid. You can get one by running:
# kafka-storage.sh random-uuid
clusterId = "xxxxxxxxxxxxxxxxxxxxxx";
formatLogDirs = true;
settings = {
listeners = [
"PLAINTEXT://:9092"
"CONTROLLER://:9093"
];
# Adapt depending on your security constraints
"listener.security.protocol.map" = [
"PLAINTEXT:PLAINTEXT"
"CONTROLLER:PLAINTEXT"
];
"controller.quorum.voters" = [
"1@127.0.0.1:9093"
];
"controller.listener.names" = ["CONTROLLER"];
"node.id" = 1;
"process.roles" = ["broker" "controller"];
# I prefer to use this directory, because /tmp may be erased
"log.dirs" = ["/var/lib/apache-kafka"];
"offsets.topic.replication.factor" = 1;
"transaction.state.log.replication.factor" = 1;
"transaction.state.log.min.isr" = 1;
};
};
# Set this so that systemd automatically create /var/lib/apache-kafka
# with the right permissions
systemd.services.apache-kafka.unitConfig.StateDirectory = "apache-kafka";
3 Likes
Thank you!
When I first built it, it failed to start the service. According to
journalctl -u apache-kafka.service
my version expected a zookeeper.connect field still.
May 07 18:48:24 nixos systemd[1]: Started Apache Kafka Daemon.
May 07 18:48:24 nixos java[1037363]: [2024-05-07 18:48:24,693] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
May 07 18:48:25 nixos java[1037363]: [2024-05-07 18:48:25,060] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
May 07 18:48:25 nixos java[1037363]: [2024-05-07 18:48:25,090] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
May 07 18:48:25 nixos java[1037363]: org.apache.kafka.common.config.ConfigException: Missing required configuration `zookeeper.connect` which has no default value.
May 07 18:48:25 nixos java[1037363]: at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2169)
May 07 18:48:25 nixos java[1037363]: at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2160)
May 07 18:48:25 nixos java[1037363]: at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1569)
May 07 18:48:25 nixos java[1037363]: at kafka.Kafka$.buildServer(Kafka.scala:72)
May 07 18:48:25 nixos java[1037363]: at kafka.Kafka$.main(Kafka.scala:91)
May 07 18:48:25 nixos java[1037363]: at kafka.Kafka.main(Kafka.scala)
May 07 18:48:25 nixos systemd[1]: apache-kafka.service: Main process exited, code=exited, status=1/FAILURE
May 07 18:48:25 nixos systemd[1]: apache-kafka.service: Failed with result 'exit-code'.
I added the default zookeeper parameter just to get it to work. Here’s my current config for reference
services.apache-kafka = {
enable = true;
clusterId = "muFUCn_OSq-6It9LTWlY1g";
formatLogDirs = true;
settings = {
"broker.id" = 0;
"log.dirs" = [ "/var/lib/apache-kafka" ];
"zookeeper.connect" = "localhost:2181";
"offsets.topic.replication.factor" = 1;
"transcation.state.log.replication.factor" = 1;
"transcation.state.log.min.isr" = 1;
listeners = [
"PLAINTEXT://:9092"
"CONTROLLER://:9093"
];
"listener.security.protocol.map" = [
"PLAINTEXT:PLAINTEXT"
"CONTROLLER:PLAINTEXT"
];
"controller.quorum.voters" = [
"1@127.0.0.1:9093"
];
"controller.listener.names" = ["CONTROLLER"];
"node.id" = 1;
"process.roles" = ["broker" "controller"];
};
};
systemd.services.apache-kafka.unitConfig.StateDirectory = "apache-kafka";
i was able to create topics. Thank you for the help! Would it be “cleaner” if I got rid of zookeeper dependency?
That’s weird, with this config I didn’t need Zookeeper at all. You’ll have to use Kafka from NixOS 23.11 I think, the one from 23.05 is too old for this.
In my case I thought it was better to get rid of Zookeeper, one less services that runs, and the fact that it used port 8080 by default was annoying.