Sometimes you may receive notifications that you are sending more data than your plan allows. If this is unexpected you might want to look a bit more closely at the data you're sending, to see if there's additional data such as headers that are not required. We don't want you sending more that data than is necessary!

Tip: Elasticsearch does perform compression on the indexes after a period of time, so this can result in the usage figures on the statistics page changing within a 24hr period.

Break out your data

If you have a single field that contains an entire JSON request for example, you can parse this data into separate fields that contain just the values you require. Then you can discard the large request field, reducing the total data sent. 

In the example below we are matching a value in the message data with a specific pattern. If we find a value that matches the pattern we create a field called Duration  with this value, all unmatched message data will be discarded.

filter {
     grok {
       match => {
         "message" => "Duration: %{NUMBER:duration}"
       }
     }
   }


Remove Headers

Headers can often contain a large amount of data, they are added automatically by Logstash when sending data to Elasticsearch. If you don't require this, you can remove them by adding the following to your Logstash filter before sending to Elasticsearch.

mutate {    
    remove_field => ["headers"]
  }


What's next?

Did this answer your question?