{"id":646,"date":"2024-07-02T12:26:07","date_gmt":"2024-07-02T12:26:07","guid":{"rendered":"https:\/\/zahiralam.com\/blog\/?p=646"},"modified":"2024-07-17T10:27:02","modified_gmt":"2024-07-17T10:27:02","slug":"how-to-install-and-set-up-apache-kafka-on-ubuntu-a-step-by-step-guide","status":"publish","type":"post","link":"https:\/\/zahiralam.com\/blog\/how-to-install-and-set-up-apache-kafka-on-ubuntu-a-step-by-step-guide\/","title":{"rendered":"How to Install and Set Up Apache Kafka on Ubuntu 24.04: A Step-by-Step Guide"},"content":{"rendered":"\n<p>Apache Kafka is a powerful distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. In this guide, we will walk you through the step-by-step process of installing and setting up Apache Kafka on Ubuntu.\n\n\n\n<p>\n\n\n\n<h2 class=\"wp-block-heading\">Prerequisites<\/h2>\n\n\n\n<p>Before you begin, ensure that you have the following:\n\n\n\n<ul class=\"wp-block-list\">\n<li>A machine running Ubuntu (preferably the latest version).<\/li>\n\n\n\n<li>Administrative access to the machine (you can execute <code>sudo<\/code> commands).<\/li>\n<\/ul>\n\n\n\n<p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Install Java<\/h3>\n\n\n\n<p>Apache Kafka requires Java to run. You can install the default Java Runtime Environment (JRE) using the following commands:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-1\">sudo apt update\nsudo apt install default-jre -y<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#sudo%20apt%20update%0Asudo%20apt%20install%20default-jre%20-y\">\n                            <button class=\"copy-button\" data-label=\"sudo apt update\nsudo apt install default-jre -y\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>Verify the installation by checking the Java version:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-2\">java -version<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#java%20-version\">\n                            <button class=\"copy-button\" data-label=\"java -version\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Download Apache Kafka<\/h3>\n\n\n\n<p>Visit the <a href=\"https:\/\/kafka.apache.org\/downloads\" target=\"_blank\" rel=\"noreferrer noopener\">official Kafka download page<\/a> to get the latest stable version. Alternatively, use the <code>wget<\/code> command to download Kafka directly:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-3\">wget https:\/\/downloads.apache.org\/kafka\/3.7.1\/kafka_2.13-3.7.1.tgz<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#wget%20https%3A%2F%2Fdownloads.apache.org%2Fkafka%2F3.7.1%2Fkafka_2.13-3.7.1.tgz\">\n                            <button class=\"copy-button\" data-label=\"wget https:\/\/downloads.apache.org\/kafka\/3.7.1\/kafka_2.13-3.7.1.tgz\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Extract Kafka<\/h3>\n\n\n\n<p>Extract the downloaded Kafka archive:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-4\">tar -xzf kafka_2.13-3.7.1.tgz<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#tar%20-xzf%20kafka_2.13-3.7.1.tgz\">\n                            <button class=\"copy-button\" data-label=\"tar -xzf kafka_2.13-3.7.1.tgz\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>Navigate to the Kafka directory:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-5\">cd kafka_2.13-3.7.1<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#cd%20kafka_2.13-3.7.1\">\n                            <button class=\"copy-button\" data-label=\"cd kafka_2.13-3.7.1\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Start the Kafka Server<\/h3>\n\n\n\n<p>Kafka requires ZooKeeper to manage and coordinate the Kafka brokers. Start the ZooKeeper service and Kafka server in separate terminal windows or tabs.\n\n\n\n<p>1. Start ZooKeeper:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-6\">bin\/zookeeper-server-start.sh config\/zookeeper.properties<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fzookeeper-server-start.sh%20config%2Fzookeeper.properties\">\n                            <button class=\"copy-button\" data-label=\"bin\/zookeeper-server-start.sh config\/zookeeper.properties\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>2. Start the Kafka server:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-7\">bin\/kafka-server-start.sh config\/server.properties<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fkafka-server-start.sh%20config%2Fserver.properties\">\n                            <button class=\"copy-button\" data-label=\"bin\/kafka-server-start.sh config\/server.properties\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Create a Kafka Topic<\/h3>\n\n\n\n<p>In a new terminal window or tab, create a topic named &#8220;my_topic&#8221; with a replication factor of 1 and a partition count of 1:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-8\">bin\/kafka-topics.sh --create --topic my_topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fkafka-topics.sh%20--create%20--topic%20my_topic%20--bootstrap-server%20localhost%3A9092%20--replication-factor%201%20--partitions%201\">\n                            <button class=\"copy-button\" data-label=\"bin\/kafka-topics.sh --create --topic my_topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Step 6: Test Apache Kafka<\/h3>\n\n\n\n<p>To ensure Kafka is working correctly, create a producer to send messages to the &#8220;my_topic&#8221; topic and a consumer to read messages from the topic.\n\n\n\n<p>1. Start a Kafka producer in a new terminal window or tab:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-9\">bin\/kafka-console-producer.sh --topic my_topic --bootstrap-server localhost:9092<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fkafka-console-producer.sh%20--topic%20my_topic%20--bootstrap-server%20localhost%3A9092\">\n                            <button class=\"copy-button\" data-label=\"bin\/kafka-console-producer.sh --topic my_topic --bootstrap-server localhost:9092\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>Type messages in the producer terminal. These messages will be sent to Kafka.\n\n\n\n<p>2. Start a Kafka consumer in another terminal window or tab:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-10\">bin\/kafka-console-consumer.sh --topic my_topic --from-beginning --bootstrap-server localhost:9092<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fkafka-console-consumer.sh%20--topic%20my_topic%20--from-beginning%20--bootstrap-server%20localhost%3A9092\">\n                            <button class=\"copy-button\" data-label=\"bin\/kafka-console-consumer.sh --topic my_topic --from-beginning --bootstrap-server localhost:9092\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>The consumer will display messages sent by the producer.\n\n\n\n<p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently Asked Questions (FAQs) <\/h2>\n\n\n\n<h4 class=\"wp-block-heading\">1. <strong>What is Apache Kafka?<\/strong><\/h4>\n\n\n\n<p>Apache Kafka is a distributed event streaming platform capable of handling high-throughput, low-latency data feeds. It is commonly used for building real-time data pipelines and streaming applications.\n\n\n\n<h4 class=\"wp-block-heading\">2. <strong>Why do I need Java to run Apache Kafka?<\/strong><\/h4>\n\n\n\n<p>Apache Kafka is written in Java and requires a Java Runtime Environment (JRE) to run. Installing Java provides the necessary environment for Kafka to operate.\n\n\n\n<h4 class=\"wp-block-heading\">3. <strong>How can I verify if Java is installed correctly on my Ubuntu machine?<\/strong><\/h4>\n\n\n\n<p>You can verify the Java installation by running the following command:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-11\">java -version<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#java%20-version\">\n                            <button class=\"copy-button\" data-label=\"java -version\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>This command should display the installed Java version.\n\n\n\n<h4 class=\"wp-block-heading\">4. <strong>Where can I download the latest version of Apache Kafka?<\/strong><\/h4>\n\n\n\n<p>You can download the latest stable version of Apache Kafka from the <a href=\"https:\/\/kafka.apache.org\/downloads\">official Kafka download page<\/a>.\n\n\n\n<h4 class=\"wp-block-heading\">5. <strong>What is ZooKeeper, and why is it needed for Kafka?<\/strong><\/h4>\n\n\n\n<p>ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. Kafka uses ZooKeeper to manage and coordinate the Kafka brokers.\n\n\n\n<h4 class=\"wp-block-heading\">6. <strong>How do I start ZooKeeper and Kafka on Ubuntu?<\/strong><\/h4>\n\n\n\n<p>To start ZooKeeper:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-12\">bin\/zookeeper-server-start.sh config\/zookeeper.properties<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fzookeeper-server-start.sh%20config%2Fzookeeper.properties\">\n                            <button class=\"copy-button\" data-label=\"bin\/zookeeper-server-start.sh config\/zookeeper.properties\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>To start Kafka:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-13\">bin\/kafka-server-start.sh config\/server.properties<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fkafka-server-start.sh%20config%2Fserver.properties\">\n                            <button class=\"copy-button\" data-label=\"bin\/kafka-server-start.sh config\/server.properties\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>These commands should be run in separate terminal windows or tabs.\n\n\n\n<h4 class=\"wp-block-heading\">7. <strong>What is a Kafka topic?<\/strong><\/h4>\n\n\n\n<p>A Kafka topic is a category or feed name to which records are stored and published. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.\n\n\n\n<h4 class=\"wp-block-heading\">8. <strong>How can I create a topic in Kafka?<\/strong><\/h4>\n\n\n\n<p>You can create a topic named &#8220;my_topic&#8221; with the following command:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-14\">bin\/kafka-topics.sh --create --topic my_topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fkafka-topics.sh%20--create%20--topic%20my_topic%20--bootstrap-server%20localhost%3A9092%20--replication-factor%201%20--partitions%201\">\n                            <button class=\"copy-button\" data-label=\"bin\/kafka-topics.sh --create --topic my_topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<h4 class=\"wp-block-heading\">9. <strong>How do I test if Kafka is working correctly?<\/strong><\/h4>\n\n\n\n<p>To test Kafka, you can create a producer to send messages to a topic and a consumer to read messages from that topic.\n\n\n\n<p>Start a Kafka producer:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-15\">bin\/kafka-console-producer.sh --topic my_topic --bootstrap-server localhost:9092<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fkafka-console-producer.sh%20--topic%20my_topic%20--bootstrap-server%20localhost%3A9092\">\n                            <button class=\"copy-button\" data-label=\"bin\/kafka-console-producer.sh --topic my_topic --bootstrap-server localhost:9092\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>Start a Kafka consumer:\n\n\n\n<div class=\"code-block-container\">\n                        <pre class=\"wp-block-code\"><code id=\"code-16\">bin\/kafka-console-consumer.sh --topic my_topic --from-beginning --bootstrap-server localhost:9092<\/code><\/pre>\n                        <amp-iframe sandbox=\"allow-scripts\" width=\"94\" height=\"72\" frameborder=\"0\" \n                                    src=\"https:\/\/zahiralam.com\/blog\/wp-content\/plugins\/amp-copy-code-button\/copier.html#bin%2Fkafka-console-consumer.sh%20--topic%20my_topic%20--from-beginning%20--bootstrap-server%20localhost%3A9092\">\n                            <button class=\"copy-button\" data-label=\"bin\/kafka-console-consumer.sh --topic my_topic --from-beginning --bootstrap-server localhost:9092\"  placeholder disabled>Copy<\/button>\n                        <\/amp-iframe>\n                    <\/div>\n\n\n\n<p>Messages typed into the producer terminal should appear in the consumer terminal.\n\n\n\n<h4 class=\"wp-block-heading\">10. <strong>Can I run Kafka on Windows or macOS?<\/strong><\/h4>\n\n\n\n<p>Yes, Apache Kafka can run on Windows, macOS, and Linux. The installation steps vary slightly for different operating systems, so refer to the <a href=\"https:\/\/kafka.apache.org\/documentation\/\">official Kafka documentation<\/a> for instructions specific to your OS.\n\n\n\n<h4 class=\"wp-block-heading\">11. <strong>What should I do if Kafka fails to start?<\/strong><\/h4>\n\n\n\n<p>If Kafka fails to start, check the following:\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ensure that ZooKeeper is running.<\/li>\n\n\n\n<li>Verify the configuration files (<code>zookeeper.properties<\/code> and <code>server.properties<\/code>) are correct.<\/li>\n\n\n\n<li>Check for any error messages in the terminal or Kafka logs.<\/li>\n<\/ul>\n\n\n\n<p>\n\n\n\n<h4 class=\"wp-block-heading\">12. <strong>How do I stop the Kafka server and ZooKeeper?<\/strong><\/h4>\n\n\n\n<p>To stop the Kafka server, press <code>Ctrl+C<\/code> in the terminal where it is running. To stop ZooKeeper, press <code>Ctrl+C<\/code> in the terminal where it is running.\n\n\n\n<h4 class=\"wp-block-heading\">13. <strong>Where can I find more information and support for Kafka?<\/strong><\/h4>\n\n\n\n<p>For more information, refer to the <a href=\"https:\/\/kafka.apache.org\/documentation\/\" target=\"_blank\" rel=\"noreferrer noopener\">official Kafka documentation<\/a>. You can also seek support from the Kafka community through forums, mailing lists, and Stack Overflow.\n\n\n\n<p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>Congratulations! You have successfully installed and set up Apache Kafka on your Ubuntu machine. You can now leverage Kafka&#8217;s powerful features to build real-time data pipelines and streaming applications. For more advanced configurations and usage, refer to the <a href=\"https:\/\/kafka.apache.org\/documentation\/\">official Kafka documentation<\/a>.\n","protected":false},"excerpt":{"rendered":"<p>Apache Kafka is a powerful distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. [&#8230;]<\/p>\n","protected":false},"author":1,"featured_media":717,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[198],"tags":[194,196,197,18,66,195],"class_list":["post-646","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-software-development","tag-apache-kafka","tag-distributed-systems","tag-event-streaming","tag-ubuntu","tag-ubuntu-24-04","tag-zookeeper"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/posts\/646","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/comments?post=646"}],"version-history":[{"count":5,"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/posts\/646\/revisions"}],"predecessor-version":[{"id":653,"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/posts\/646\/revisions\/653"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/media\/717"}],"wp:attachment":[{"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/media?parent=646"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/categories?post=646"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zahiralam.com\/blog\/wp-json\/wp\/v2\/tags?post=646"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}