push to kafka with mysql data using python and python kafka client sdk

 https://medium.com/@sangeethaprabhagaran/pushing-mysql-data-into-kafka-6ab8ab5bf755

kafka does not support http request

hence any kafka push pull need to go through developed SDK or u make an SDK based on kafka documentaion on your own

push to kafka using python sdk from mysql db example:

Prerequisites:-

• Install MySQL on your laptop, create an employee table, and import data into it. You can refer to my other post
• Install Kafka on your laptop, and create an employee topic. You can refer to my other post
• Install the MySQL connector by executing the below command in your command prompt —

pip install mysql-connector-python

• Once the above installation successful, then install the kafka python by executing the below command in your command prompt —

pip install kafka-python

• As a next step you can verify these installs in your python shell as below

Execution:

• We will try to read the MySQL table data in to cursor convert it in to Json format and send the records to the Kafka topic over a loop. We will also import dumps from the Json library, so that we can serialize our value (records) into Json.

• Initialize the connection to your MySQL database — development. You can use your username and password here.

• Set the cursor to fetch all the records from the employee table —

• Create an instance of the KafkaProducer client with the basic configurations. Please ensure to start your zookeeper and Kafka server before executing this code. We are trying to connect to our local Kafka server and setting a JSON serializer for value.In Kafka data is stored as key, value pairs. However, having key is optional. So we are skipping the key here. Also, there are many other serializers available. Please refer to Kafka-python to get more options.

• Next step is to send the MySQL records in a JSON format to the Kafka topic — employee, we have created already

• Once you execute the below code you will see the below output, which are the records from the MySQL employee table with serialization applied.

• To verify the records you can run a Kafka consumer as below

bin\windows\kafka-console-consumer.bat — bootstrap-server localhost:9092 — topic employee — from-beginning

I had run the push multiple times and hence you see duplicate messages here. Also, we have used “from-beginning” option to read all the messages. In case we want to look at only the latest messages, then we can skip this option -

That’s it. We have pushed the MySQL data into the Kafka topic. You can access this complete code in my github