Setting up, configuring and sending logs and metrics to Logstash can appear to be a complicated process, this guide aims to show you that it using Logit really is that simple.
Where to start...
Filebeat is the most popular tool for ingesting logs and metrics because it works with minimal configuration and with a wide range of common log formats. For this reason we recommend it as a good place to start.
Tip: Remember, Filebeat doesn't cover all logging scenarios and that is why there are several different shippers available. If you don't think that Filebeat is the solution that meets your needs then please take a look at other options in our Data Source Wizard.
After you have created your first ELK stack you will see it on your Dashboard. Under the Actions section, click on the "Add Source" button within this section to get started.
Next select "Both (Logs and Metrics)" and press the next button. Select "Shippers" and press the "Search for datasource" button.
You can see a list of all of the shippers in the results section. As explained earlier, we are going to use the recommended Filebeat in our example so click Filebeat to continue.
Tip: If you do require a shipper other than Filebeat, this is where you choose it. Choosing a shipper will show you the source wizard that explains how to get up and running with your chosen data source.
The first section of the wizard contains instructions for installing Filebeat. The instructions cover several OS so please follow the instructions for the OS that you are using. There are instructions for installing Filebeat on Debian, Ubuntu, Mint, CentOS, RHEL, Fedora, MacOS and Windows. However, if you are using a different OS to the ones listed, try the link at the bottom for the downloads page to see if you can find what you require.
Next you need to locate the configuration file, the wizard shows you where to find this file for each type of OS. Click the "next step" button to do this.
Configuring the inputs
We need to open the configuration file and make some changes. These changes are also described in the source wizard. Search for the filebeat.inputs section. You will see the following two lines in the file:
# Change to true to enable this input configuration.
Change the enabled to true. It should now look as follows:
# Change to true to enable this input configuration.
You will also see the following lines in the filebeat.inputs section:
# Paths that should be crawled and fetched. Glob based paths.
This is how you tell Filebeat where the files that you wish to ingest are located. Filebeat is currently looking in the directory /var/log/ for files. The Asterisk means it is looking for all files in this directory with the .log extension but it is possible to be more specific and actually name the files here if this is what you want to do. Change this to point to where your log files are located.
Let's now skip forward to the configure output section of the wizard. Click the "Next step" button until you arrive at this section.
In this example we are sending data to Logstash rather than Elasticsearch so we must comment out all of the lines in the configuration file that relate to Elasticsearch. Commenting out is done using the hash symbol. The Elasticsearch output section of the configuration file should look as follows:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
# protocol: "https"
# username: "elastic"
# password: "changeme"
Now we need to update the Logstash output section of the configuration file to include your settings. You will notice that the wizard actually displays your unique settings to save you looking them up.
Currently the hosts line in the configuration file will look as follows:
You need to remove the hash and then copy over the localhost part with the settings shown in the wizard so that it looks similar to below:
You then need to add the following two lines to the section:
When finished, the Logstash output section of the configuration file should look similar to the following:
# The Logstash hosts
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
# Certificate for SSL client authentication
# Client Certificate Key
Now we are ready to validate the file. Click "Next step" to do this.
This section shows us how to check that the updates we have made to the configuration file during the configuration process haven't broken the format. Again there are instructions for several OS's so follow the instructions for the one that you are using. If the validation fails then go back to the Configuring Inputs and Configuring Outputs sections and check that your file is in the same format as the example shown.
When the validation has passed we are ready to start ingesting data. Click the "Next step" button to learn how to start doing this.
There are instructions that show how to start Filebeat for each type of OS. Follow the instructions for the system that you are using.
Confirm data is in Logstash
Having followed the instructions we should now be able to view the logs. To do this navigate back to the Dashboard. This time under Actions for your Stack, press the "Kibana" button.
Tip: Kibana is the tool that is used to visualise your data, we can use it to see any logs and metrics that have been sent to Logstash on this Stack. We have written a guide that explains how to get the most out of Kibana here.
When Kibana is launched you should be able to see the logs and metrics that were ingested into Elasticsearch by Logstash. It is possible to filter this data by time, fields and content etc.
If you don't see any logs or metrics in Kibana then there a few things that you should check.
- Are there any files in the folder that was added during file inputs configuration?
- Is the format of the configuation file correct? Does it validate?
- Do the Logstash settings in the file match the settings in the source wizard? Have they been copied correctly?
- Using index patterns to search your logs and metrics with Kibana
- Getting Started - Viewing your Logs and Metrics
- How do I send data from Filebeat to Logstash
- How can I diagnose no data appearing in Elasticsearch and Kibana