This repo contain resources to publish application logs to Kafka topic

A) Kafka Resources:

1. https://docs.confluent.io/current/quickstart/cos-docker-quickstart.html #Kafka cli is command line kafka tool used to publish and consume messages
2. https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.0/kafka_2.12-2.4.0.tgz 3)https://docs.spring.io/spring-boot/docs/current/reference/pdf/spring-boot-reference.pdf

B) FluentD resources:

1. https://docs.fluentd.org/input/tail

2. https://docs.fluentd.org/v/0.12/output/kafka #Configuration link below is used to define configuration file used by fluentd
3. https://docs.fluentd.org/output/kafka#example-configuration #Install Fluentd on mac osx
4. https://support.treasuredata.com/hc/en-us/articles/207048557-Installing-td-agent-on-MacOS-X-

C) Other resources:

1. /~https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/fluentd #Log messages from http input plugin
2. https://docs.fluentd.org/input/http #Use the below link to launch all the confluent kafka services
3. /~https://github.com/confluentinc/examples/tree/5.3.1-post/cp-all-in-one

/~https://github.com/CDCgov/openshift-fluentd-forwarder/blob/master/fluentd-forwarder-template.yaml

Step 1: Install docker on your laptop a) Windows: https://docs.docker.com/docker-for-windows/install/ Mac: https://docs.docker.com/docker-for-mac/install/

Step 2: Setup local kafka environment using confluent gitrepo, Go through the link A1, Once setup is done run "docker ps" it should show all the kafka services are running

Step 3: create a kafka topic called "messages" using kafka cli Download Kafka cli using A2 Extract the .tgz file and navigate to root directory then run below command to create kafka topic "messages"

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic message

Step 4: Test kafka environment by opening two terminal windows one for consumer and one for producer In producer terminal window: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic messages In Consumer terminal window: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic messages --from-beginning Now What ever message you are entering in producer window will appear in consumer window

Step 5: Using B4 install fluentd on system Dy default the fluentd will provide conf file with few source and match tags Run "sudo launchctl load /Library/LaunchDaemons/td-agent.plist" in new terminal
Run "less /var/log/td-agent/td-agent.log" should show the fluentd services started with default config

Step 6: Stop the fluentd services using below command "sudo launchctl unload /Library/LaunchDaemons/td-agent.plist" Now it's time to update the configuration located in "/etc/td-agent/td-agent.conf"

Replace the content with below, you may need to change config based on your requirement using B3

brokers localhost:9092 use_event_time true

default_topic messages

required_acks -1 compression_codec gzip

Step 7: Restart the fluentd services using "sudo launchctl load /Library/LaunchDaemons/td-agent.plist"

Step 8:Assuming nodejs app and kafka consumer and producer services are still running curl -d '{"name":"value1", "email":"value2"}' -H "Content-Type: application/json" -X POST http://localhost:4000/submit-data

In Kafka consumer terminal you should be able to see the message received

Steps to execute nodejs application:

Step1: npm install Step2: node app.js

curl -d '{"name":"value1", "email":"value2"}' -H "Content-Type: application/json" -X POST http://localhost:4000/submit-data