How do I check if a field is aggregatable?
To check if a field is aggregatable open OpenSearch choose the Management option on the left-hand side menu and then choose index patterns. If there is a check/circle under the aggregatable column then this means the field is aggregated.
How do I make a field aggregatable?
1. Select a field
First, you need to choose a field. For this example I've opted to use the message field, as you can see in the image below the message field is not aggregated.
Now you need to find out what index pattern this particular message field appears under. As you're on the index pattern page you'll be able to view the index pattern at the top of the page. The Logstash-* index pattern is being used in the example above.
2. Fetching the templates
The second part requires the use of the 'OpenSearch Dev Tools' section, select the wrench tool icon on the OpenSearch menu and run the following command in the dev tools section. You can also use the OpenSearch API to get the templates, but in this example you'll be using 'OpenSearch Dev Tools'.
GET _cat/templates
This will return the names of the templates currently in OpenSearch and OpenSearch Dashboards. The name of the template required in the example is Logstash. So now on a new line in OpenSearch Dev Tools you're going to run the following command.
GET _template/logstash
Running this command will fetch the Logstash index template which can be seen below.
{
"order" : 0,
"version" : 60001,
"index_patterns" : [
"logstash-*"
],
"settings" : {
"index" : {
"refresh_interval" : "5s"
}
},
"mappings" : {
"_default_" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
}
}
},
{
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword"
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo_point"
},
"latitude" : {
"type" : "half_float"
},
"longitude" : {
"type" : "half_float"
}
}
}
}
}
},
"aliases" : { }
}
3. Creating a keyword field
In the template above find the "message_field" section. You can see that the current mapping type is text, you can't aggregate on a text field type. You need a keyword field type in order to aggregate.
The easiest way to change the mapping type of the field is to input a new template. Copy the template above into a text editor and convert the "message_field" to a keyword. You can do this by adding the next code snippet to the message field.
"fields" : {
"keyword" : {
"ignore_above" : 2048,
"type" : "keyword"
}
}
The message field mapping should now look like this:
"mappings" : {
"_default_" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 2048
}
}
}
}
},
(mappings __default__ has been deprecated in version 7 and will appear as mappings _doc)
"mappings" : {
"_doc" : {
"dynamic_templates" : [
You need to set a reasonable size on 'ignore above' so not to impact performance. Most messages are smaller than 1500 characters, but they may grow. In this example you have allowed up to 2048 total characters, longer messages will be ignored.
4. Changing the template order
You also need to change the order of the template to make sure it gets applied to the message field. If you change the order number to 4 it will have the highest order number out of all of the templates. This means it will be the top-level template that gets applied. The way orders work for templates is that the lowest order number gets applied first and the higher number sits on top of that. So a template with an order number 0 would be applied first, then an order number of 1 would be applied second, an order number of 4 would sit above both of those and be applied as the top-level template. Go ahead and change the order to have the value of 4.
PUT _template/logstash_msg_keyword
{
"order" : 4,
"version" : 60001,
"index_patterns" : [
"logstash-*"
5. Putting a new template into OpenSearch
Copy the new template as it is ready to be added. Before you paste the template into OpenSearch. You need to first add the following line into 'OpenSearch Dev Tools.' This tells OpenSearch you're going to input a new template and name it logstash_msg_keyword.
PUT _template/logstash_msg_keyword
On a new line in 'OpenSearch Dev Tools' paste the new template. You should have something similar to the below section.
PUT _template/logstash_msg_keyword
{
"order" : 4,
"version" : 60001,
"index_patterns" : [
"logstash-*"
],
"settings" : {
"index" : {
"refresh_interval" : "5s"
}
},
"mappings" : {
"_default_" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 2048
}
}
}
}
},
{
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword"
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo_point"
},
"latitude" : {
"type" : "half_float"
},
"longitude" : {
"type" : "half_float"
}
}
}
}
}
},
"aliases" : { }
}
6. Send test data with a date in the future
Templates are applied on index creation, as data is likely to already have been sent to the stack today. You need to use a future index creation. This can be achieved by using a date in the future for example 1/1/2030. You can then use your stacks API key to send data from the command line into the stack. This can be found on the Stack > Settings page of your Logit.io Dashboard.
curl -i -H "ApiKey: <your-stack-api-key" -i -H "Content-Type: application/json" -H "LogType: default" https://api.logit.io/v2 -d '{"message":"Logit.io Test Log", "@timestamp":"2019-11-08T11:00:0.000Z"}'
7. Confirm the results
Once you have confirmed data has arrived into your stack. On OpenSearch you need to go to the Management tab, select index patterns, refresh the index field list and search for the message.keyword field, check whether the field is now aggregatable.
What's next
Learn how to start viewing your Logs and Metrics
Access your stack via the OpenSearch API