What is Logtrail?
LogTrail is a plugin for Kibana to view, analyse, search and tail log events from multiple hosts in real-time. LogTrail makes use of a live tail view that makes reading logs easy.
Updating the default Logtrail configuration
To update the Logtrail configuration file on your stack, you can use the Logit Elasticsearch API to create an index for Logtrail. All you need to do is send in your Logtrail configuration to the Logit Elasticsearch API as a JSON file or message.
Depending on whether your stack is in ApiKey or Basic Auth Mode will decide your next step. You can check this by choosing View Stack Settings > Elasticsearch Settings.
ApiKey Authorisation Mode
To do this you will need your Elasticsearch endpoint address and your ApiKey. These can be accessed from your dashboard by choosing View Stack Settings > Elasticsearch Settings..
To put your logtrail.json configuration file into the Elasticsearch index you would run a command similar to below. Remember to substitute in your endpoint address and ApiKey.
curl -XPUT 'https://<stack-id>-es.logit.io/.logtrail/config/1?apikey=<stack-apikey>&pretty=true' -H 'Content-Type: application/json' -d@<path_to_logtrail.json>
To retrieve your current config using the Elasticsearch API you can run a command similar to the below.
curl -XGET 'https://<stack-id>-es.logit.io/.logtrail/config/1?apikey=<stack-apikey>&pretty=true'
Basic Authorisation Mode
To do this you will need your Elasticsearch endpoint address, username and password. These can be accessed from your dashboard by choosing View Stack Settings > Elasticsearch Settings.
You will need to encode your username and password to base64encoding. A REST client can generate this for you using your credentials.
# The base64encoding needs to be in the following format:
username:password
To put your logtrail.json configuration file into the Elasticsearch index using basic authorisation mode you would run a command similar to below. Remember to substitute in your endpoint address and base64encoded username and password.
curl -i -XPUT -H "Authorization:Basic [BASE64EncodedCredentials]" "https://<stack-id>-es.logit.io/.logtrail/config/1?pretty" -H 'Content-Type: application/json' -d@<path_to_logtrail.json>
To retrieve your current config using basic auth mode you can run a command similar to the below.
curl -XGET -H "Authorization:Basic [BASE64EncodedCredentials]" "https://<stack-id>-es.logit.io/.logtrail/config/1?pretty"
Understanding the Logtrail configuration
Let's take a look at an example.
{
"version" : 2,
"index_patterns" : [
{
"es": {
"default_index": "*-*"
},
"tail_interval_in_seconds": 10,
"es_index_time_offset_in_seconds": 0,
"display_timezone": "local",
"display_timestamp_format": "MMM DD HH:mm:ss",
"max_buckets": 500,
"default_time_range_in_days" : 0,
"max_hosts": 100,
"max_events_to_keep_in_viewer": 5000,
"default_search": "",
"fields" : {
"mapping" : {
"timestamp" : "@timestamp",
"hostname" : "beat.hostname",
"program": "source",
"message": "message"
},
"message_format": "{{{message}}}",
"keyword_suffix" : "keyword"
},
"color_mapping" : {
}
}
]
}The default index is the Elasticsearch Index where the events are stored. In the example above the default index is *-* .
default_search if specified will apply a default search text while launching Logtrail. The value can be any search word/term.
One example could be 'ssh' - which shows all logs where ssh is in the message field.
A second example would be 'log.level:severe' this would show all logs where log.level field equals severe.
If specifying a default_search value the field name should be a valid field in Elasticsearch.
Fields - Edit this to map different event fields in Elasticsearch to Logtrail fields.
Hostname - hostname of where the events were received. hostname needs to be a keyword field. Use agent.hostname for version 7 ELK and beat.hostname for version 6 ELK. (Unless you have specified something else)
Program - the program that generated the event.
Message - actual message event that will be used by search.
Example - Mappings
For a version 7 stack running Logtrail the maps should be similar to below:
"mapping" : {
"timestamp" : "@timestamp",
"hostname" : "agent.hostname",
"program": "source",
"message": "message"
}For a version 6 stack running Logtrail the maps should be similar to below:
"mapping" : {
"timestamp" : "@timestamp",
"hostname" : "beat.hostname",
"program": "source",
"message": "message"
}
Example - message_format
You can add additional fields to the Logtrail log event using message_format. This will replace whatever is defined in fields.mapping.
If you wanted to add host and pid to the log event you would do so by adding the following to message_format field.
"fields" : {
"mapping" : {
"timestamp" : "@timestamp",
"hostname" : "hostname",
"program": "program",
"message": "syslog_message"
},
"message_format": "{{{host}}} | {{{pid}}} : {{{syslog_message}}}"
}Things to consider. Logtrail uses Handlerbar templates to replace the fields in message_format with actual values.
{{{field_name}}} will add field with hyperlink.
{{field_name}} will add field without hyperlink.
{{[nested.field]}} should be used to add nested fields.
Example - Colour Mappings
You can colour code messages by specifying fields and field_values.
An example on how to do this. The chosen field is log.level and the field values are notice, warn, error and emerg. Use hex colour values to give them their respective colours.
"color_mapping" : {
"field": "log.level",
"mapping": {
"notice": "#3e66ed",
"warn": "#17b6a7",
"error": "#b41f41",
"emerg": "#9a85c6"
}
}If there are no matches in the mapping, the default colour ( as per CSS ) will be applied.
Some Example Configurations
Stack Version - 6.8.4
{
"version" : 2,
"index_patterns" : [
{
"es": {
"default_index": "filebeat-*"
},
"tail_interval_in_seconds": 10,
"es_index_time_offset_in_seconds": 0,
"display_timezone": "local",
"display_timestamp_format": "MMM DD HH:mm:ss",
"max_buckets": 500,
"default_time_range_in_days" : 0,
"max_hosts": 100,
"max_events_to_keep_in_viewer": 5000,
"default_search": "",
"fields" : {
"mapping" : {
"timestamp" : "@timestamp",
"hostname" : "beat.hostname",
"program": "source",
"message": "message"
},
"message_format": "{{{message}}}",
"keyword_suffix" : "keyword"
},
"color_mapping" : {
}
},
{
"es": {
"default_index": "logstash-*"
},
"tail_interval_in_seconds": 10,
"es_index_time_offset_in_seconds": 0,
"display_timezone": "local",
"display_timestamp_format": "MMM DD HH:mm:ss",
"max_buckets": 500,
"default_time_range_in_days" : 0,
"max_hosts": 100,
"max_events_to_keep_in_viewer": 5000,
"default_search": "",
"fields" : {
"mapping" : {
"timestamp" : "@timestamp",
"hostname" : "host",
"program": "source",
"message": "message"
},
"message_format": "{{{message}}}",
"keyword_suffix" : "keyword"
},
"color_mapping" : {}
}
]
}
Stack Version - 7.3.2
{
"version" : 2,
"index_patterns" : [
{
"es": {
"default_index": "filebeat-*"
},
"tail_interval_in_seconds": 10,
"es_index_time_offset_in_seconds": 0,
"display_timezone": "local",
"display_timestamp_format": "MMM DD HH:mm:ss",
"max_buckets": 500,
"default_time_range_in_days" : 0,
"max_hosts": 100,
"max_events_to_keep_in_viewer": 5000,
"default_search": "",
"fields" : {
"mapping" : {
"timestamp" : "@timestamp",
"hostname" : "agent.hostname",
"program": "source",
"message": "message"
},
"message_format": "{{{message}}}",
"keyword_suffix" : "keyword"
},
"color_mapping" : {}
}
]
}