Forwarding IBM MQ Messages to Kafka using Kafka Connect

This is a quick guide to demonstrate how to usekafka-connect-mq-source that compliments the IBM MQ Tutorial.

Prerequisites: Docker

Run MQ Server in Docker

1. Get the latest image of MQ server from Docker Hub. Verify by docker image.

docker pull ibmcom/mq:latest

2. Run the MQ server. Once it’s running, verify by docker ps.

Queue manager: QM1
Queue: DEV.QUEUE.1
Channel: DEV.APP.SVRCONN
Listener DEV.LISTENER.TCP on port 1414

# Create a volume to persist the data even if container is deleted.
docker volume create qm1data
# Create a network.
docker network create mq-demo-network
# Run the image with the following settings.
docker run --env LICENSE=accept --env MQ_QMGR_NAME=QM1 --volume qm1data:/mnt/mqm --publish 1414:1414 --publish 9443:9443 --network mq-demo-network --network-alias qmgr --detach --env MQ_APP_PASSWORD=password ibmcom/mq:latest

Run Apache Kafka

1. Download from https://kafka.apache.org/downloads (any version since 1.0.0 should do), then extract it.

tar -xzf kafka_2.11-2.1.0.tgz

2. In one terminal, go to the Kafka root directory and run Zookeeper.

cd kafka_2.11-2.1.0
bin/zookeeper-server-start.sh config/zookeeper.properties

3. In another terminal, go to the Kafka root directory and run Kafka server.

cd kafka_2.11-2.1.0
bin/kafka-server-start.sh config/server.properties

4. In a third terminal, go to the Kafka root directory and run a Kafka consumer to listen from a TSOURCE topic (we’ll create it shortly).

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TSOURCE --from-beginning

Note: On Windows platforms use bin\windows\ instead of bin/, and change the script extension to .bat.

Run MQ Source Connector

1. Clone and build the connector.

git clone https://github.com/ibm-messaging/kafka-connect-mq-source.git
cd kafka-connect-mq-source
mvn clean package

2. Copy the connector configuration file to your home directory to edit it safely.

cp config/mq-source.properties ~

3. Configure the properties according to our MQ setup.

Note: remember to uncomment mq.user.name and mq.password properties.

mq.queue.manager=QM1
mq.connection.name.list=localhost(1414)
mq.channel.name=DEV.APP.SVRCONN
mq.queue=DEV.QUEUE.1
mq.user.name=app
mq.password=password
topic=TSOURCE

4. Run the connector

Note: Replace <connector-root-directory and <version> by yourself.

CLASSPATH=<connector-root-directory>/target/kafka-connect-mq-source-<version>-jar-with-dependencies.jar bin/connect-standalone.sh config/connect-standalone.properties ~/mq-source.properties

Run MQ Client in Docker

1. Download Dockerfile from here.
2. Build and run it.

Notice that we are connecting it to the same network as MQ server, mq-demo-network.

docker build -t mq-demo .
docker run --network mq-demo-network -ti mq-demo

3. The application will prompt you to enter some settings. Just press return to skip the first 6 questions; enter password for the last one.

1. Type the name of your queue manager (leave blank for QM1):

2. Type the host name or IP address for your queue manager:
qmgr
3. Type the listener port for your queue manager (leave blank for 1414):

4. Type the name of the application channel (leave blank for 'DEV.APP.SVRCONN'):

5. Type the name of the queue (leave blank for 'DEV.QUEUE.1'):

6. Type the application user name (leave blank for 'app'):

7. Type the password for this user:
password

4. When it is up, you will see:

Connecting to queue manager 'QM1'
at address 'mq-demo-network(1414)'
through channel 'DEV.APP.SVRCONN'
as user 'app'
. . .
Connected!

Type PUT [number] to put a given number of messages to the queue
Type GET [number] to get a given number of messages to the queue
Type AUTO [seconds] to put and get messages automatically for a given number of seconds
Type EXIT to quit
Type HELP to display this message

Send message to MQ and receive in Kafka

In the MQ Client terminal, run put <n> to put n number of messages to the DEV.QUEUE.1 queue.

$ put 20
Attempting to put 20 messages to the queue...
Successfully put 20 messages
Check the MQ Console to see that these 20 messages have been added to the queue

In the Kafka consumer terminal, you shall see messages coming into the TSOURCE topic. TA DA ~

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TSOURCE --from-beginning
Message #0��������
Message #1��������
Message #2��������
Message #3��������
Message #4��������
Message #5��������
Message #6��������
Message #7��������
Message #8��������
Message #9��������
Message #10��������
Message #11��������
Message #12��������
Message #13��������
Message #14��������
Message #15��������
Message #16��������
Message #17��������
Message #18��������
Message #19��������