Start Design Your Online Store

Calendar is loading...
- Available
- Booked
- Pending
Whats App

Building custom search architecture for your site using ELK: Logstash

Yes Soft

Logging Pipeline

This section is talking about installing the following components:

  1. Kafka: This API is a queuing API
  2. Logstash: This API has some jobs to do with logs
  3. ElasticSearch: Search API, Similar in job to SQL but radically different API.
  4. Kibana: To Interact with ElasticSearch
  5. Grafana: To Draw the data.

Note: Don’t use spaces in Kafka directory.


  1. JDK v12
  2. Node.JS for Kibana

Installing The Materials

  1. Kafka: Install it from . we’re using kafka_2.12-2.3.0
  2. Logstash: Install from . We’re using logstash-7.2.0
  3. Elastic Search: Install from . We’re using elasticsearch-7.1.1
  4. Kibana: Install from  . We’re using kibana-7.1.1-windows-x86_64

Configuring Zookeeper (Kafka)

Check The following Lines in: config/


And on config/ the following:




Starting Zookeeper

Run the following command while you are on Kafka files directory:

./bin/windows/zookeeper-server-start.bat ./config/

It should start to show some messages, that’s how you know it’s running:

Starting Kafka Server

Run the following command while you are on Kafka files directory:

./bin/windows/kafka-server-start.bat ./config/

Again, some messages means that the thing actually work.

Configuring Elasticsearch

Make sure the following lines are not commented on ./config/elasticsearch.yml Yes-Soft-App TEST-NODE-1

http.port: 9200

Running Elasticsearch Engine

Run the following command while you are In Elastic Search Files Directory:


Some messages and it’s ok.

Logstash Configuration and Start

In your Logstash Directory create the following:


Inside this add the following:

input {

   kafka {

       bootstrap_servers => “localhost:9092”

       topics => [“yes”]



filter {

 json {

   source => “message”



output {

   elasticsearch {

       hosts => [“localhost:9200”]

       index => “yes_final”

       workers => 1


   stdout { 

       codec => rubydebug 



Test the config using this command: ./logstash.bat -t -f logstash-kafka.conf

There will be some warnings BUT you should be able to see the status as OK

Then start the logstash using the command: ./logstash.bat -f logstash-kafka.conf

Finally Kibana

Change the following file config/kibana.yml and uncomment the following:

server.port: 5601 localhost

elasticsearch.hosts: [“http://localhost:9200”]

Then start Kibana using the command: ./bin/kibana.bat

This will take a while But when it’s running it’s Finally Over.

Testing the Setup

In Kafka Directory Write the following command in Terminal:

./bin/windows/kafka-console-producer.bat –broker-list localhost:9092 –topic yes_topic

And write a message

Open another tab and write the command:

./bin/windows/kafka-console-consumer.bat –bootstrap-server localhost:9092 –topic yes_topic –from-beginning

You should be able to see the message here.

Now, Open the Kibana Website — localhost:5601 — go to developer tools — from the sidebar — and then write the following command:

GET /yes_topic/_search


 “query”: {

    “match_all”: {}



And watch the messages rolling in the second output screen.

Add Topic

Run the command:

./bin/windows/kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic mohammad

You can see the code At Yes-Soft Github

Categories : Blog, e-commerce, ISHTAR
Tags: , , ,

User Avatar
Yes Soft ( Yes Soft )

There Is No Biography

: 27
: Yes Soft
« »

1 Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Copyright © 2022 Yes Soft - All right reserved.
Implemented By Yes Soft Team.