What is an ingest pipeline?
An ingest pipeline allows you to change and structure your data before indexing. For example, you could use a pipeline to remove a field, extract a value from some text or enhance parts of your data.
How do I use the Logit.io ingest pipelines?
Logit.io ingest pipelines already come pre-configured in your Stack. So it's as simple as using Filebeat to ship your data.
You need to enable the module for the data you are trying to send, details of how to do this can be found in your chosen source integration.
Configure the output of Filebeat to your hosted Logstash instance and you’re good to go.
Once your data arrives at Elasticsearch you will be able to view the pre-configured dashboards associated with your data. Once you have sent data, your dashboards can be found in Kibana.
To view your dashboards launch your Kibana instance and navigate to Dashboards in the left-hand menu.
For example, if you had the system module enabled and were sending logs with Filebeat you would see dashboards similar to the below.
How to stop a pipeline from being run?
To stop an ingest pipeline from being run/used, you can simply comment out the name of the pipeline in the Logstash Pipeline Editor section.
To do this use the 'Configure Logstash Pipelines' link on the Stack Dashboard or by navigating to Logstash Pipelines from the Stack Settings page.
Now you need to find the name of the ingest pipeline you want to remove or disable and comment it out using #.
For example, if you didn't want to use the haproxy-log ingest pipeline you would do:
else if /apache-error/ in [@metadata][pipeline] {
mutate {
add_field => { "[@metadata][logit_pipeline]" => "apache-error" }
}
}
# else if /haproxy-log/ in [@metadata][pipeline] {
# mutate {
# add_field => { "[@metadata][logit_pipeline]" => "haproxy-log" }
# }
# }
else if /nginx-access/ in [@metadata][pipeline] {
mutate {
add_field => { "[@metadata][logit_pipeline]" => "nginx-access" }
}
}
Alternatively, you can also delete the Ingest Pipeline you don't wish to use from Elasticsearch.
To do this launch your Kibana instance and navigate to Dev Tools in the left-hand menu.
From here you can run a GET request to see which pipelines you have installed.
GET _ingest/pipeline/
Now you need to choose the name of the pipeline you wish to delete and run a DELETE request.
DELETE _ingest/pipeline/haproxy-log