A Flatline alert matches when the total number of events is under a given
threshold for a time period. The threshold is the minimum number of events for an alert not to be triggered therefore the time period must contain less than the
threshold value e.g.
Flatline type - Match when there are less than X events in Y time
If you have an alert to check the frequency of events of a heartbeat message that drops below 100 in ten minutes you would see the alert is triggered. Here is an example below:
name: "Heartbeat flatline Example Alert"
query: "message: heartbeat*"
Using the example above you can ensure that you have a working flatline alert. To have a working alert rule you will need the following:
nameof the alert
indexthat you need to check against
timeframeyou want to set and
queryyou are checking against.
Additional options that you may want to include in your alert rule are explained below.
use_terms_query is an additional option, which if you set to true then ElastAlert will make an aggregation query against Elasticsearch to get counts of documents matching each unique value of the
query_key which can be set similarly to the
query looks like this:
query_key: "message: heartbeat*"
This means that an alert will be triggered if any value of the
query_key has been seen at least once and then falls below the threshold.
If your timeframe is 10 minutes and you look for events that happened in the last 10 minutes,
timeframe_elapsed will always default to False.
For flatline rules,
first_event gets populated with an empty "placeholder" timestamp after the first query is made. So, it's guaranteed to get populated after the first query.
Make sure to proofread the rule you have written to ensure that it is what you expect to see as most of the issues regarding alerts not working correctly are related to the points above. Below are some examples of common issues:
Issues with YAML spacing/formatting
Required fields for rules that are missing
Two rules with the same name
Read Logit's introduction to alerting
Learn more about ElastAlert rules: if you'd like to learn more about ElastAlert rules from the people that built it, check out their cheat sheet.