kafka 101

 kafaka 101

https://www.youtube.com/watch?v=ncTosfaZ5cQ

https://github.com/marcel-dempers/docker-development-youtube-series/blob/master/messaging/kafka/README.md

kafka uses zookeeper to manage topics and partitions(a topic can be divide into paritions (each broker can hold same amount of parition for replications)

so message from partiions of 1 broker can be replicated to parition of another broker of the same topic(cluster)

you can start consumer producer and create topic anywhere u have kafka installed(including zookeeper) :

Create the Topic:

docker exec -it zookeeper-1 bash

/kafka/bin/kafka-topics.sh \

--create \

--zookeeper zookeeper-1:2181 \

--replication-factor 1 \

--partitions 3 \

--topic Orders

Describe our Topic:

/kafka/bin/kafka-topics.sh \

--describe \

--topic Orders \

--zookeeper zookeeper-1:2181

consumer 

docker exec -it zookeeper-1 bash

/kafka/bin/kafka-console-consumer.sh \

--bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092 \

--topic Orders --from-beginning

With a consumer in place, we can start producing messages

producer

docker exec -it zookeeper-1 bash

echo "New Order: 1" | \

/kafka/bin/kafka-console-producer.sh \

--broker-list kafka-1:9092,kafka-2:9092,kafka-3:9092 \

--topic Orders > /dev/null

create/describe topic:

zookeepr

https://dattell.com/data-architecture-blog/what-is-zookeeper-how-does-it-support-kafka/#:~:text=ZooKeeper%20is%20used%20in%20distributed,of%20Kafka%20topics%20and%20messages.

ZooKeeper is primarily used to track the status of nodes in the Kafka cluster and maintain a list of Kafka topics and messages.

new version of kafaka ask consumer to specify broker (like this example, before it asks to specify zookeeper, its the same idea)

https://stackoverflow.com/questions/58255240/kafkawhy-topic-creation-in-kafka-is-done-in-zookeper-host-instead-of-broker

https://stackoverflow.com/questions/43563977/can-a-kafka-producer-create-topics-and-partitions

kafka producer has a property

so if producing a message to a topic does not exist, topic can be auto created

The kafka broker has a property: auto.create.topics.enable

If you set that to true if the producer publishes a message to the topic with the new topic name it will automatically create a topic for you.

The Confluent Team recommends not doing this because the explosion of topics, depending on your environment can become unwieldy, and the topic creation will always have the same defaults when created. It's important to have a replication-factor of at least 3 to ensure durability of your topics in the event of disk failure.