What is Logtrail?

LogTrail is a plugin for Kibana to view, analyse, search and tail log events from multiple hosts in realtime. LogTrail makes use of a live tail view that makes reading logs easy.

Updating the default Logtrail configuration

To update the Logtrail configuration file on your stack, you can use the Logit Elasticsearch API to create an index for Logtrail. All you need to do is send in your Logtrail configuration to the Logit Elasticsearch API as a json file or message.

Dependant on whether your stack is in ApiKey or Basic Auth Mode will decide your next step. You can check this by choosing Stack Settings > Elasticsearch.

ApiKey Authorisation Mode

To do this you will need your Elasticsearch endpoint address and your ApiKey. These can be accessed from your dashboard by choosing Stack Settings > Elasticsearch.

To put your logtrail.json configuration file into the Elasticsearch index you would run a command similar to below. Remembering to substitute in your endpoint address and ApiKey.

curl -XPUT 'https://<stack-id>-es.logit.io/.logtrail/config/1?apikey=<stack-apikey>&pretty=true' -H 'Content-Type: application/json' -d@<path_to_logtrail.json>

To retrieve your current config using the Elasticsearch API you can run a command similar to the below.

curl -XGET 'https://<stack-id>-es.logit.io/.logtrail/config/1?apikey=<stack-apikey>&pretty=true' 

Basic Authorisation Mode

To do this you will need your Elasticsearch endpoint address, username and password. These can be accessed from your dashboard by choosing Stack Settings > Elasticsearch.
You will need to encode your username and password to base64encoding.  A rest client can generate this for you using your credentials.

# The base64encoding needs to be in the following format:
  username:password

To put your logtrail.json configuration file into the Elasticsearch index using basic authorisation mode you would run a command similar to below. Remembering to substitute in your endpoint address and base64encoded username and password.

curl -i -XPUT -H "Authorization:Basic [BASE64EncodedCredentials]" "https://<stack-id>-es.logit.io/.logtrail/config/1?pretty" -H 'Content-Type: application/json' -d@<path_to_logtrail.json>

To retrieve your current config using basic auth mode you can run a command similar to the below.

curl -XGET -H "Authorization:Basic [BASE64EncodedCredentials]" "https://<stack-id>-es.logit.io/.logtrail/config/1?pretty"

Understanding the Logtrail configuration

Let's take a look at an example.

{
  "version" : 2,
  "index_patterns" : [
    {      
      "es": {
        "default_index": "*-*"
      },
      "tail_interval_in_seconds": 10,
      "es_index_time_offset_in_seconds": 0,
      "display_timezone": "local",
      "display_timestamp_format": "MMM DD HH:mm:ss",
      "max_buckets": 500,
      "default_time_range_in_days" : 0,
      "max_hosts": 100,
      "max_events_to_keep_in_viewer": 5000,
      "default_search": "",
      "fields" : {
        "mapping" : {
            "timestamp" : "@timestamp",
            "hostname" : "beat.hostname",
            "program": "source",
            "message": "message"
        },
        "message_format": "{{{message}}}",
        "keyword_suffix" : "keyword"
      },
      "color_mapping" : {
      }
    }
  ]
}

The default index is the Elasticsearch Index where the events are stored.  In the example above the default index is *-* .

default_search if specified will apply a default search text while launching Logtrail. The value can be any search word/term. 

  • One example could be 'ssh' - shows all logs where ssh is in the message field. 
  • A second example would be 'log.level:severe' this would show all logs where log.level field equals severe. 

If specifying a default_search value the field name should be a valid field in Elasticsearch.

Fields - Edit this to map different event fields in Elasticsearch to Logtrail fields.

Hostname - hostname of where the events were received. hostname needs to be be a keyword field. Use agent.hostname for version 7 ELK and beat.hostname for version 6 ELK. (Unless you have specified something else)

Program - the program that generated the event.

Message - actual message event that will be used by search.

Example - Mappings

  • For a version 7 stack running Logtrail the maps should be similar to below:
"mapping" : {
    "timestamp" : "@timestamp",
    "hostname" : "agent.hostname",
    "program": "source",
    "message": "message"
}
  • For a version 6 stack running Logtrail the maps should be similar to below:
"mapping" : {
    "timestamp" : "@timestamp",
    "hostname" : "beat.hostname",
    "program": "source",
    "message": "message"
}

Example - message_format

You can add additional fields to the Logtrail log event using message_format. This will replace whatever is defined in fields.mapping.

If you wanted to add host and pid to the log event you would do so by adding the following to message_format field.

"fields" : {
    "mapping" : {
        "timestamp" : "@timestamp",
        "hostname" : "hostname",
        "program": "program",
        "message": "syslog_message"
    },
    "message_format": "{{{host}}} | {{{pid}}} : {{{syslog_message}}}"
  }

Things to consider. Logtrail uses Handlerbar templates to replace the fields in message_format with actual values. 

  • {{{field_name}}} will add field with hyperlink. 
  • {{field_name}} will add field without hyperlink. 
  • {{[nested.field]}} should be used to add nested fields.

Example - Colour Mappings

You can colour code messages by specifying fields and field_values.
An example on how to do this. The chosen field is log.level and the field values are notice, warn, error and emerg. Use hex colour values to give them their respective colours.

"color_mapping" : {
  "field": "log.level",
  "mapping": {
    "notice": "#3e66ed",
    "warn": "#17b6a7",
    "error": "#b41f41",
    "emerg": "#9a85c6"
  }
}

If there are no matches in the mapping, the default colour ( as per CSS ) will be applied.


Some Example Configurations

Stack Version - 6.8.4

{
  "version" : 2,
  "index_patterns" : [
    {      
      "es": {
        "default_index": "filebeat-*"
      },
      "tail_interval_in_seconds": 10,
      "es_index_time_offset_in_seconds": 0,
      "display_timezone": "local",
      "display_timestamp_format": "MMM DD HH:mm:ss",
      "max_buckets": 500,
      "default_time_range_in_days" : 0,
      "max_hosts": 100,
      "max_events_to_keep_in_viewer": 5000,
      "default_search": "",
      "fields" : {
        "mapping" : {
            "timestamp" : "@timestamp",
            "hostname" : "beat.hostname",
            "program": "source",
            "message": "message"
        },
        "message_format": "{{{message}}}",
        "keyword_suffix" : "keyword"
      },
      "color_mapping" : {
      }
    },
    {      
      "es": {
        "default_index": "logstash-*"
      },
      "tail_interval_in_seconds": 10,
      "es_index_time_offset_in_seconds": 0,
      "display_timezone": "local",
      "display_timestamp_format": "MMM DD HH:mm:ss",
      "max_buckets": 500,
      "default_time_range_in_days" : 0,
      "max_hosts": 100,
      "max_events_to_keep_in_viewer": 5000,
      "default_search": "",
      "fields" : {
        "mapping" : {
            "timestamp" : "@timestamp",
            "hostname" : "host",
            "program": "source",
            "message": "message"
        },
        "message_format": "{{{message}}}",
        "keyword_suffix" : "keyword"
      },
      "color_mapping" : {}
  }
]
}

Stack Version - 7.3.2

{
  "version" : 2,
  "index_patterns" : [
    {      
      "es": {
        "default_index": "filebeat-*"
      },
      "tail_interval_in_seconds": 10,
      "es_index_time_offset_in_seconds": 0,
      "display_timezone": "local",
      "display_timestamp_format": "MMM DD HH:mm:ss",
      "max_buckets": 500,
      "default_time_range_in_days" : 0,
      "max_hosts": 100,
      "max_events_to_keep_in_viewer": 5000,
      "default_search": "",
      "fields" : {
        "mapping" : {
            "timestamp" : "@timestamp",
            "hostname" : "agent.hostname",
            "program": "source",
            "message": "message"
        },
        "message_format": "{{{message}}}",
        "keyword_suffix" : "keyword"
      },
      "color_mapping" : {}
  }
]
}

What's Next?

Did this answer your question?