A Flatline alert matches when the total number of events is under a given threshold for a time period. The threshold is the minimum number of events for an alert not to be triggered therefore the time period must contain less than the threshold value e.g.

  • Flatline type - Match when there are less than X events in Y time

If you have an alert to check the frequency of events of a heartbeat message that drops below 100 in ten minutes you would see the alert being triggered. Here is an example below:

name: "Heartbeat flatline"
type: flatline
index: "*-*"
threshold: 100
timeframe:
  minutes: 10
filter:
- query:
    query_string:
      query: "message: heartbeat*"
use_count_query: true
doc_type: _doc
alert:
  - email
email:
  - test@logit.io

Using the example above you can ensure that you have a working flatline alert. To have a working alert rule you will need the following:

  • The name of the alert 
  • The type set to flatline 
  • The index that you need to check against 
  • The threshold  you require 
  • The timeframe  you want to set and 
  • The query you are checking against.


Additional Options

Additional options that you may want to include in your alert rule are explained below.

use_terms_query is an additional option, which if you set to true then ElastAlert will make an aggregation query against Elasticsearch to get counts of documents matching each unique value of the query_key

The query_key which can be set similarly to the query looks like this: 

query_key: "message: heartbeat*"

This means that an alert will be triggered if any value of the query_key has been seen at least once and then falls below the threshold.

If your timeframe is 10 minutes and you look for events that happened in the last 10 minutes, timeframe_elapsed will always default to False.

For flatline rules, first_event gets populated with an empty "placeholder" timestamp after the first query is made. So, it's guaranteed to get populated after the first query.


Troubleshooting

Make sure to proof read the rule you have written to ensure that it is what you expect to see as most of the issues regarding alerts not working correctly are related to the points above. Below are some examples of common issues:

  • Issues with yaml spacing / formatting
  • Incorrect syntax 
  • Required fields for rules that are missing
  • Two rules with the same name

What's next?

Did this answer your question?