Managing the collection, storage, and visualization of infrastructure metrics might seem like a complex undertaking. By harnessing the power of Victoria Metrics, Grafana, and the remote write capability, you'll not only streamline metric collection but also gain valuable insights into your infrastructure's performance. So, let's embark on a journey to simplify infrastructure monitoring and empower your organization with data-driven decisions. This guide aims to show you that by using Logit.io it is actually a simple process.

In this guide, we'll walk you through the process of using a remote write URL to send system metrics to an Ingestion Pipeline via Telegraf. You'll learn how to customize the data scraping, understand how the metrics are stored in Victoria Metrics, and use Grafana to explore and interpret your valuable data.

Tip: Remember, System metrics via Telegraf doesn't cover all the scenarios and that is why there are several different shippers available. If you don't think that System Metrics via Telegraf is the solution that meets your needs then please take a look at other options in our Data Source Integrations.

Telegraf is a versatile agent for collecting and reporting metrics and data. It's commonly used to capture system performance metrics, making it an ideal choice for sending these metrics to Victoria Metrics. By combining Telegraf with Victoria Metrics and Grafana, you can centralize your infrastructure monitoring and gain deeper insights into your systems.

After you have created your first Logit.io stack you will see it on your Dashboard, click View Data Integrations to get started.

As mentioned earlier, we are going to use the System Metrics via Telegraf source in our example so search for or select System Metrics via Telegraf to get started.

Tip: If you do require a different data source, this is where you choose it. Choosing a data source will show you the source wizard instructions that explain how to get up and running with your chosen data source.

The first section of the wizard contains instructions for installing Telegraf. The instructions cover several OS so please follow the instructions for the OS that you are using. There are instructions for installing Telegraf on Windows, MacOS, Ubuntu/Debian, RedHat and CentOS, SLES/openSUSE and FreeBSD/PC-BSD.

However, if you are using a different OS to the ones listed, try the link at the bottom to read more about how to configure data scraping and configuration options for Telegraf.

Next, you need to locate the configuration file, the next step in the wizard shows you where to find this file. The configuration file is pre-configured to scrape the system metrics from your hosts, add the configuration code to the configuration file from the previous step.

We need to open the configuration file and make some changes. These changes are also described in the source wizard. Find the configuration file normally located at /etc/telegraf/telegraf.conf. The exact location for your OS will be detailed in Step 1 of the source wizard.

Copy the entire contents from step 2 as shown below using the Copy button.

Then in your Telegraf configuration file paste the copied content at the bottom of the file, save and close the file.

Let's now skip forward to the configure output plugin section of the wizard. Move down to Step 3 to continue.

After updating the configuration file inputs, the next step is to configure the output plug-in, enabling Telegraf to seamlessly transmit your data to Logit.io in the Prometheus format. This can be achieved by incorporating the code snippet from Step 3 of the source wizard into your configuration file, just as we did for the Inputs above.

Tip: The contents of the output code snippet in this step are pre-configured with your Logit.io stack id remote write url, so should look similar to that shown below. If it’s not pre-configured your details you can find your Remote Write Url by going to Dashboard > Infrastructure Metrics > Manage Stack > Remote Write.

Copy the entire contents from step 3 as shown below using the Copy button.

Then in your Telegraf configuration file paste the copied content at the bottom of the file, save and close the file.

Now we are ready to start Telegraf and start shipping metrics to Logit.io.

There are instructions that show how to start Telegraf for each type of OS in Step 4 of the source wizard. Follow the instructions for the system that you are using.

Having followed the instructions we should now be able to view the metrics in Grafana. To do this choose Launch Dashboard in Step 5 of the source wizard or from your main dashboard choose Launch Metrics.

When Grafana is launched you should be able to see the metrics that were ingested into OpenSearch via the ingestion pipeline. It is possible to filter this data by time, fields, and content, so feel free to explore!

If you don't see any metrics in Grafana then there are a few things that you should check.

While setting up the Telegraf for system metrics monitoring, you may encounter issues with data visibility. If your collected data is not appearing as expected in Victoria Metrics or Grafana, consider the following troubleshooting steps:

Telegraf Configuration Validation:

Syntax Errors: Double-check your Telegraf configuration file for syntax errors or typos. A minor mistake can prevent Telegraf from collecting and transmitting data.

Output Plugin Configuration:

Output Plugin Settings: Review the configuration of the Telegraf output plugin responsible for sending data to Victoria Metrics. Ensure that the remote write URL is correctly configured.

Firewall and Network Issues:

Firewall Rules: Verify that your firewall settings allow outbound connections from the host running Telegraf. Check for any network issues that might be preventing data transmission. This help article contains some useful firewall commands that may help.

Remote Write URL:

Remote Write URL Validity: Confirm that the remote write URL specified in Telegraf matches the one provided. A mismatched URL can prevent data from reaching the correct destination.