Logging Pipeline

This section is talking about installing the following components:

1. Kafka: This API is a queuing API
2. Logstash: This API has some jobs to do with logs
3. ElasticSearch: Search API, Similar in job to SQL but radically different API.
4. Kibana: To Interact with ElasticSearch
5. Grafana: To Draw the data.

Note: Don’t use spaces in Kafka directory.

Requirements

1. JDK v12
2. Node.JS for Kibana

Installing The Materials

1. Kafka: Install it from https://kafka.apache.org/ . we’re using kafka_2.12-2.3.0
2. Logstash: Install from https://elasic.co/ . We’re using logstash-7.2.0
3. Elastic Search: Install from https://elastic.co/ . We’re using elasticsearch-7.1.1
4. Kibana: Install from https://elastic.co/  . We’re using kibana-7.1.1-windows-x86_64

Configuring Zookeeper (Kafka)

Check The following Lines in: config/zookeeper.properties

clientPort=2181

And on config/server.properties the following:

broker.id=0

listeners=PLAINTEXT://:9092

dataDir=Logz

zookeeper.connect=localhost:2181

Starting Zookeeper

Run the following command while you are on Kafka files directory:

./bin/windows/zookeeper-server-start.bat ./config/zookeeper.properties

It should start to show some messages, that’s how you know it’s running:

Starting Kafka Server

Run the following command while you are on Kafka files directory:

./bin/windows/kafka-server-start.bat ./config/server.properties

Again, some messages means that the thing actually work.

Configuring Elasticsearch

Make sure the following lines are not commented on ./config/elasticsearch.yml

cluster.name: Yes-Soft-App

node.name: TEST-NODE-1

#network.host: 0.0.0.0

http.port: 9200

Running Elasticsearch Engine

Run the following command while you are In Elastic Search Files Directory:

./bin/elasticsearch.bat

Some messages and it’s ok.

Logstash Configuration and Start

In your Logstash Directory create the following:

/bin/logstash-kafka.conf

Inside this add the following:

input {

   kafka {

       bootstrap_servers => “localhost:9092”

       topics => [“yes”]

   }

}

filter {

 json {

   source => “message”

 }

}

output {

   elasticsearch {

       hosts => [“localhost:9200”]

       index => “yes_final”

       workers => 1

   }

   stdout { 

       codec => rubydebug 

   }

}

Test the config using this command: ./logstash.bat -t -f logstash-kafka.conf

There will be some warnings BUT you should be able to see the status as OK

Then start the logstash using the command: ./logstash.bat -f logstash-kafka.conf

Finally Kibana

Change the following file config/kibana.yml and uncomment the following:

server.port: 5601

server.host: localhost

elasticsearch.hosts: [“http://localhost:9200”]

Then start Kibana using the command: ./bin/kibana.bat

This will take a while But when it’s running it’s Finally Over.

Testing the Setup

In Kafka Directory Write the following command in Terminal:

./bin/windows/kafka-console-producer.bat –broker-list localhost:9092 –topic yes_topic

And write a message

Open another tab and write the command:

./bin/windows/kafka-console-consumer.bat –bootstrap-server localhost:9092 –topic yes_topic –from-beginning

You should be able to see the message here.

Now, Open the Kibana Website — localhost:5601 — go to developer tools — from the sidebar — and then write the following command:

GET /yes_topic/_search

{

 “query”: {

    “match_all”: {}

 }

}

And watch the messages rolling in the second output screen.

Add Topic

Run the command:

./bin/windows/kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic mohammad

You can see the code At Yes-Soft Github