Required Windows executables:
- zookeeper-3.4.8
- kafka_2.11
- elasticsearch-2.3.3
- logstash-2.3.2
- kibana-4.5.1-windows
Introduction
This article helps kickoff beginner to start working on ELK with basic/default configurations, i.e., Elasticsearch 2.3.3 + Logstash 2.3.2 + Kibana 4.5. It is mainly for making above things run on Windows 7 + platform and some tricks to make it work. I am putting together ways to solve the common issues that I faced while configuring the above things.
Background
You can get more information on architecture and working of Kafka ELK on respective official ebsites. Google for it. . For quickly knowing about these terms, you can treat each of them as:
- Zookeeper: Kind of server which keeps track of Topics, Clusters, Message indices
- Kafka: A log server
- Elasticsearch: Your fast searcher
- Logstash: Mediator between Kafka logs and Elasticsearch logs
- Kibana: Reporting on visualization server/tool
Using the code/batch scripts for configuring Kafka ELK on Windows 7 onward:
For editing files, I prefer Notepad++.
You need the below zip files extracted to get started configuring each of the below:
- jre1.8.0_92 or jre1.8.0_92 depending on 32/64 bit architecture. I am using 64 bit machine.
- Zookeeper 3.4.8
- Kafka 2.11
- Elasticsearch 2.3.3
- Logstash 2.3.2
- kibana-4.5.1-windows
I have extracted them on D:. You can choose one from your drive. But remember to change on every place of code where it appears to be D: with one that you have selected.
1) JRE
Install Java and set environment variable:
JAVA_HOME = C:\Program Files\Java\jre1.8.0_92
Those who are new to environment variables can Google how to set environment variables.
While setting environment variable under Path field, add the path to Java runtime. C:\Program Files\Java\jre1.8.0_92
2) Zookeeper
//Rename D:\zookeeper-3.4.8\conf\zookeeper.cfg file to D:\zookeeper-3.4.8\conf\zoo.cfg, edit the respective line to set data directory as:
dataDir=D:/zookeeper-3.4.8/data
To start zookeeper, open new Command window and execute:
D:\zookeeper-3.4.8>bin>zkserver
3) Kafka
To start Kafka server, open new Command window and execute:
D:\kafka_2.11-0.10.0.0>.\bin\windows\kafka-server-start.bat .\config\server.properties
To create a topic in Kafka, open new Command Window and execute the command below. You may create many topics, but go with the one below topic name 'test'.
D:\kafka_2.11-0.10.0.0>.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic test
To create a Producer, open new command window and execute:
D:\kafka_2.11-0.10.0.0>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
In Producer window, you can type messages to be logged by Kafka. These messages are later used by Logstash to push them to Elasticsearch logs.
4) Elasticsearch
To configure elasticsearch with defaults, open new command window and execute:
D:\elasticsearch\bin>elasticsearch
To install elasticsearch service, open new command window and execute:
D:\elasticsearch\bin>service install
To start elasticsearch service, execute:
D:\elasticsearch\bin>service start
If you are unable to install elasticsearch service, you can try the trick below:
- Extract old version of elasticsearch 1.4.4 on D:\
- Delete Bin, Config, Lib folders from elasticsearch 1.4.4
- Paste Bin, Config, Lib folders from elasticsearch 2.3..3 into elasticsearch 1.4.4 folder.
- Run the below commands again. It should successfully install and start elasticsearch service.
D:\elasticsearch\bin>elasticsearch
D:\elasticsearch\bin>service install
D:\elasticsearch\bin>service start
Default Url for elasticsearch will be:
http://localhost:9200
Tip: For testing purposes, you can use Fiddler.exe to post json request to the above url and get your search result.
5) Logstash
For using Logstash to write to elasticsearch logs, open new command window again and execute:
D:\logstash-2.3.2\bin>logstash -e "input { kafka { topic_id => 'test' } }
output { stdout{ } elasticsearch { hosts => localhost } }"
If you get the error stating "The signal HUP is used by JVM...
", there is a trick you can do.
Try installing logstash plugin by executing the command:
D:\logstash-2.3.2\bin>logstash-plugin.bat install logstash-output-kafka-master\logstash-output-kafka
Then, execute the original command:
D:\logstash-2.3.2\bin>logstash -e "input { kafka { topic_id => 'test' } }
output { stdout{ } elasticsearch { hosts => localhost } }"
It should start you logstash, that will get kafka messages posted to elasticsearch.
6) Kibana
To start Kibana server, open new command window and execute:
D:\kibana-4.5.1-windows\bin>kibana
You will get message kibana server running at http://0.0.0.0:5601.
Use actual url to access Kibana: http://localhost:5601
Use logstash-*
in kibana to list all logs.
YOU ARE NOW SET TO CREATE YOUR FIRST KIBANA VISUALIZATION/REPORT.
Points of Interest
Configure all the applications to work together and learn reporting, analyzing, visualizing data using Kibana.