When transporting data from a source to your Logit stacks using Logstash, there may be fields you do not wish to retain or see in Kibana. You can remove these using the mutate  filter plugin. There are several different ways of using this plugin to cover a wide range of use-cases, and so it is important to choose the right strategy depending on your situation.

If you always want to remove a field, or fields, from your data regardless of the situation, you can include the remove_field  setting. This setting should point to an array of field names (or a single field name). :

filter {
    mutate {    
        remove_field => ["field_name"]
    }
}


If the array only contains one field name, you can omit the square brackets:

remove_field => "field_name"


If you want to remove more than one field you can supply additional field names to the array:

filter {
    mutate {    
        remove_field => [ "%{@index}","%{@version}","%{@type}" ]
    }
}

Conditional Removals

Often you only want to remove a field if a given condition is true. One of the most common use-cases for a conditional remove is to only remove a field if the field's value matches another value:

if [action] == "login" {
    mutate {
        remove_field => ["last_login"]
    }
}


You can invert these conditions to remove a field if the condition is not true by using the not equals comparison operator (!=) :

if [action] != "login"


To remove a field if the field's value makes up part of a larger string (i.e. a sub-string) you can use the special in  keyword:

if [action] in "login" {
    mutate {
        remove_field => ["last_login"]
    }
}


You can also use the in  keyword to check if a field's value matches at least one of many strings in an array:

if [action] in ["hello", "world", "foo"] {
    mutate {
        remove_field => ["field_name"]  
    }
}


These types of conditions can be inverted using the not  operator:

if [action] not in ["hello", "world", "foo"]

Removing by range

If your field's value is an integer, you can check if the value is inside a range of values. Before doing this, you must ensure that the value is interpreted as an integer by Logstash and not a number inside a string. To do this, use the convert plugin:

mutate {
    convert => [ "field_name", "integer" ]


Then you will be able to use a compound statement to compare the value against a minimum and maximum value to check if the value is in range:

if [field_name] > 50 and [field_name] < 100 {
    mutate {
        remove_field => [ "field_name" ]
    }
}


The above snippet will remove the field_name  field if it's value is between 50 and 100.

What's next?

Did this answer your question?