Logging Pipeline
This section is talking about installing the following components:
- Kafka: This API is a queuing API
- Logstash: This API has some jobs to do with logs
- ElasticSearch: Search API, Similar in job to SQL but radically different API.
- Kibana: To Interact with ElasticSearch
- Grafana: To Draw the data.
Note: Don’t use spaces in Kafka directory.
Requirements
- JDK v12
- Node.JS for Kibana
Installing The Materials
- Kafka: Install it from https://kafka.apache.org/ . we’re using kafka_2.12-2.3.0
- Logstash: Install from https://elasic.co/ . We’re using logstash-7.2.0
- Elastic Search: Install from https://elastic.co/ . We’re using elasticsearch-7.1.1
- Kibana: Install from https://elastic.co/ . We’re using kibana-7.1.1-windows-x86_64
Configuring Zookeeper (Kafka)
Check The following Lines in: config/zookeeper.properties
clientPort=2181
And on config/server.properties the following:
broker.id=0
listeners=PLAINTEXT://:9092
dataDir=Logz
zookeeper.connect=localhost:2181
Starting Zookeeper
Run the following command while you are on Kafka files directory:
./bin/windows/zookeeper-server-start.bat ./config/zookeeper.properties
It should start to show some messages, that’s how you know it’s running:
Starting Kafka Server
Run the following command while you are on Kafka files directory:
./bin/windows/kafka-server-start.bat ./config/server.properties
Again, some messages means that the thing actually work.
Configuring Elasticsearch
Make sure the following lines are not commented on ./config/elasticsearch.yml
cluster.name: Yes-Soft-App
node.name: TEST-NODE-1
#network.host: 0.0.0.0
http.port: 9200
Running Elasticsearch Engine
Run the following command while you are In Elastic Search Files Directory:
./bin/elasticsearch.bat
Some messages and it’s ok.
Logstash Configuration and Start
In your Logstash Directory create the following:
/bin/logstash-kafka.conf
Inside this add the following:
input {
kafka {
bootstrap_servers => “localhost:9092”
topics => [“yes”]
}
}
filter {
json {
source => “message”
}
}
output {
elasticsearch {
hosts => [“localhost:9200”]
index => “yes_final”
workers => 1
}
stdout {
codec => rubydebug
}
}
Test the config using this command: ./logstash.bat -t -f logstash-kafka.conf
There will be some warnings BUT you should be able to see the status as OK
Then start the logstash using the command: ./logstash.bat -f logstash-kafka.conf
Finally Kibana
Change the following file config/kibana.yml and uncomment the following:
server.port: 5601
server.host: localhost
elasticsearch.hosts: [“http://localhost:9200”]
Then start Kibana using the command: ./bin/kibana.bat
This will take a while But when it’s running it’s Finally Over.
Testing the Setup
In Kafka Directory Write the following command in Terminal:
./bin/windows/kafka-console-producer.bat –broker-list localhost:9092 –topic yes_topic
And write a message
Open another tab and write the command:
./bin/windows/kafka-console-consumer.bat –bootstrap-server localhost:9092 –topic yes_topic –from-beginning
You should be able to see the message here.
Now, Open the Kibana Website — localhost:5601 — go to developer tools — from the sidebar — and then write the following command:
GET /yes_topic/_search
{
“query”: {
“match_all”: {}
}
}
And watch the messages rolling in the second output screen.
Add Topic
Run the command:
./bin/windows/kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic mohammad
You can see the code At Yes-Soft Github
Muchas gracias. ?Como puedo iniciar sesion?