How can I see the health of my stacks?

If you are experiencing performance problems shipping logs to Logit using Filebeat or other shippers it could indicate a problem with your Elasticsearch stack. In order to diagnose what the potential issue may be you can check the health of your Stack(s) via your dashboard.
 

Using the Dashboard to view the health of your Stacks

You can check the health of your Stacks from your Dashboard. You will see in the top right-hand corner of the page that there is a "Stack health status" display. The health status includes a summary of the current status of your Elasticsearch along with a percentage gauge. The given percentage is a combined figure of the overall health of all of the Stack(s) visible on your dashboard.

If there aren't currently any issues with any of your Stacks the gauge will show health as being 100%, anything less than this indicates a problem with at least one of your Stacks. A triangle warning icon appears next to any Stack that has a problem. Hovering over this icon gives more information about the issue such as whether the problem originates in Kibana, Logstash or Elasticsearch.
 

Viewing the health of your Stacks using your Api Key

You can also check the health of your stack by accessing the API directly. To do this, from your Dashboard:

  • Click the "Settings" button on the Stack that you want to investigate.
  • Select "Elasticsearch" from the menu on the left-hand side of the page. 
  • Here you will see a section called "Elastic Search Access" which contains your Endpoint Address and your API Key.

Use these to create a URL in the format shown below:

[Your Endpoint Address]/_cluster/health?apikey=[Your Api Key]
 

This displays a list of values similar to what is shown below:

{
"cluster_name" : "********-****-****-****-************",  
"status" : "green",  
"timed_out" : false,  
"number_of_nodes" : 3,  
"number_of_data_nodes" : 3,  
"active_primary_shards" : 3,  
"active_shards" : 6,  
"relocating_shards" : 0,  
"initializing_shards" : 0,  
"unassigned_shards" : 0,  
"delayed_unassigned_shards" : 0,  
"number_of_pending_tasks" : 0,  
"number_of_in_flight_fetch" : 0,  "task_max_waiting_in_queue_millis" : 0,  "active_shards_percent_as_number" : 100.0
}


The main value to be concerned with here is the "status". 

Green shows that everything is fine and your cluster is fully operational. 

Yellow shows that Elasticsearch has allocated all of the primary shards but some or all of the replicas have not been allocated - if more shards were to disappear then this may impact your data.

Red indicates that some or all of the primary shards are not currently ready - you are missing data and any searches may return partial or incomplete results.
  

Logstash logs and filters

Viewing your ELK Logstash logs is another good way to get an overview of activity on your stack, and is therefore a good place to look when trying to diagnose Logstash configuration changes that may have impacted your stack performance. To view your Logstash logs from your Dashboard:

  • Click the "Settings" button on the Stack that you want to investigate.
  • Select "Log files" from the menu on the left-hand side of the page. 


Your stack Logstash logs view gives you the ability to understand if there is a potential issue and where in the ELK pipeline the cause may potentially lie. Could there be an issue with the data that you are sending? Is there a potential problem with how the data is being received and filtered? Is there a general problem with Logstash itself?

If the the logs indicate a potential problem with Logstash and how it is processing the data that is being received, the problem may lie with your Logstash filters. If you have created custom filters it could be that they are preventing data from being processed. To view your Logstash filters from the dashboard:

  • Click the "Settings" button on the Stack that you want to investigate.
  • Select "Logstash Filters" from the menu on the left-hand side of the page. 

You can sense check your Logstash filters and make any syntax changes as necessary. All Logit stacks enable you to test your Logstash filter syntax before applying them to your stack.

At the very bottom of the page there is a section labelled "Danger Zone" with a "Restart Logstash" button. Pressing this will restart your Logstash instance. You will have to enter the name of your stack or your Stack ID to confirm that you want to go ahead with the restart process, during this time Logstash is unable to receive any logs. Depending on the way that logs are being sent to Logstash, this could result in some logs being dropped.

Rebuilding your Stack

If you think that your Elasticsearch has gone down Logit makes it easy for you to recover and rebuild the Stacks yourself.  You can rebuild any Stack that you think is causing problems or failing to respond. If, for example, you have made changes or ran some commands that may be causing intermittent issues, then rebuilding may quickly resolve the problem. You will not lose your Elasticsearch data when you rebuild the Stack, however there is a chance that any data that is currently in the process of being sent may be lost.

To rebuild a Stack, from the Dashboard:

  • Click on the Settings button. This takes you to the Stack details page.
  • From here scroll right down to the bottom of the page and you will see a section labelled "Danger Zone".
  • You will see a "Rebuild this stack" button. 

You will have to enter the name of your stack or your Stack ID to confirm that you want to go ahead with the rebuild process.
 

What's next

Did this answer your question?