It will take a few minutes for the pipeline to start streaming logs. Then, Logstash: sudo service logstash start Now that we have all the pieces in place, it’s time to start all the components in charge of running the data pipeline.įirst, we’ll start Filebeat: sudo service filebeat start replication-factor 1 -partitions 1 -topic apache Next, we’re going to create a topic for our Apache logs: bin/kafka-topics.sh -create -zookeeper localhost:2181 You should begin to see some INFO messages in your console: INFO Registered We are now ready to run kafka, which we will do with this script: sudo /opt/kafka/bin/kafka-server-start.sh Next, let’s download and extract Kafka: wget Kafka uses ZooKeeper for maintaining configuration information and synchronization so we’ll need to install ZooKeeper before setting up Kafka: sudo apt-get install zookeeperd Our last and final installation involves setting up Apache Kafka - our message broker. Otherwise, the lines are sent in JSON to Kafka. Note the use of the codec.format directive - this is to make sure the message and timestamp fields are extracted correctly. In the output section, we are telling Filebeat to forward the data to our local Kafka server and the relevant topic (to be installed in the next step). In the input section, we are telling Filebeat what logs to collect - apache access logs. Let’s open the Filebeat configuration file at: /etc/filebeat/filebeat.ymlĮnter the following configurations: filebeat.inputs: To install Filebeat, we will use: sudo apt-get install filebeat You will be presented with the Kibana home page.Īs mentioned above, we will be using Filebeat to collect the log files and forward them to Kafka. Now, we can start Kibana with: sudo service kibana start We will then open up the Kibana configuration file at: /etc/kibana/kibana.yml, and make sure we have the correct configurations defined: server.port: 5601Įlasticsearch.url: " These specific configurations tell Kibana which Elasticsearch to connect to and which port to use. As before, we will use a simple apt command to install Kibana: sudo apt-get install kibana Let’s move on to the next component in the ELK Stack - Kibana. We’re applying some filtering to the logs and we’re shipping the data to our local Elasticsearch instance. Match => Īs you can see - we’re using the Logstash Kafka input plugin to define the Kafka host and the topic we want Logstash to pull from. Also, we need to define the private IP of our EC2 instance as a master-eligible node: network.host: "localhost"Ĭluster.initial_master_nodes: [" "localhost:9092" Since we are installing Elasticsearch on AWS, we will bind Elasticsearch to localhost. Tee -a /etc/apt//elastic-7.x.listĪll that’s left to do is to update your repositories and install Elasticsearch: sudo apt-get update & sudo apt-get install elasticsearchīefore we bootstrap Elasticsearch, we need to apply some basic configurations using the Elasticsearch configuration file at: /etc/elasticsearch/elasticsearch.yml : sudo su Our next step is to add the repository definition to our system: echo "deb stable main" | sudo Since version 7.x, Elasticsearch is bundled with Java so we can jump right ahead with adding Elastic’s signing key: wget -qO - | sudo apt-key add -įor installing Elasticsearch on Debian, we also need to install the apt-transport-https package: sudo apt-get update We will start with installing the main component in the stack - Elasticsearch. The example logs used for the tutorial are Apache access logs. Finally, I added a new elastic IP address and associated it with the running instance. I started the instance in the public subnet of a VPC and then set up a security group to enable access from anywhere using SSH and TCP 5601 (for Kibana). In real-life scenarios you will probably have all these components running on separate machines. To perform the steps below, I set up a single Ubuntu 16.04 machine on AWS EC2 using local storage. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |