A Kafka Connector that replicates data between Zookeeper and Kafka
A Zookeeper Kafka connector implementation for both sink and the source
$ git clone https://github.com/s8sg/kafka-connect-zookeeper$ (cd kafka-connect-zookeeper && mvn clean package)
Run zookeeper somehow
Here are instructions on how to run it using docker.$ docker run -d --name zookeeper -p 2181:2181 confluent/zookeeper
We are going to use te same zk for Kafka and for the ZK connector
Download and run kafka 0.9.0 locally by following the instructions at http://kafka.apache.org/documentation.html#quickstart
$ curl -o kafka_2.11-0.9.0.0.tgz http://www.us.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz$ tar xvfz kafka_2.11-0.9.0.0.tgz$ cd kafka_2.11-0.9.0.0$ ./bin/kafka-server-start.sh config/server.properties
Configure The Source and Sink properties
connect-zk-source.properties
| Name | Description |
|---|---|
| zk-hosts | Single ZooKeeper Host, or Comma Seperated List of ZooKeeper Hosts in a Cluster |
| zk-nodes | Single ZooKeeper Data Node, or Comma Seperated List of Zookeeper Nodes |
| Name | Description |
|---|---|
| zk-hosts | Single ZooKeeper Host, or Comma Seperated List of ZooKeeper Hosts in a Cluster |
| zk-node | ZooKeeper Data Node |
Run your connect-zookeeper-source plugin
$ export CLASSPATH=/path/to/kafka-connect-zookeeper/target/kafka-connect-zookeeper-1.0.jar$ kafka_2.11-0.9.0.0/bin/connect-standalone.sh kafka-connect-zookeeper/connect-standalone.properties kafka-connect-zookeeper/config/connect-zk-source.properties
Write stuff to zookeeper node test-data-source (the connector will read from, as configured in connect-zk-source.properties)
You could set the data using zkcli.sh. The current repo comes with an utility (zk_util.py) to upload data to zk node
$ pip install kazoo$ python zk_util.py upload localhost:2181 /test/test-data-source this_is_a_test_data
Read the data out from the kafka topic named ‘test’ (that is the that this connector will write to, as configured in connect-zk-source.properties)
$ kafka_2.11-0.9.0.0/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test{"schema":{"type":"string","optional":false},"payload":"this_is_a_test_data"}
Run your connect-zookeeper-sink plugin
$ export CLASSPATH=/path/to/kafka-connect-zookeeper/target/kafka-connect-zookeeper-1.0.jar$ kafka_2.11-0.9.0.0/bin/connect-standalone.sh kafka-connect-zookeeper/connect-standalone.properties kafka-connect-zookeeper/config/connect-zk-sink.properties
Check that the zookeeper-sink plugin has written the data to the zookeeper node test-data-sink
$ python zk_util.py download localhost:2181 /test/test-data-sinkthis_is_a_test_data