In order to send data to your Logit stacks from an Amazon S3 Bucket you must update permissions in the AWS (Amazon Web Service) IAM Policy being used. IAM (the Identity and Access Management) is a web service that helps users securely control access to their AWS resources. Amazon IAM controls who is authenticated and authorised to use your Amazon S3 Bucket.
 

Preparing your AWS permissions

The following permission should be applied:

  • s3:ListBucket - to check if the S3 Bucket exists and list objects in it.
  • s3:GetObject - to check object metadata and download objects from S3 Buckets.

You permissions should appear as follows:

{

    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SidID",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
             ],
            "Resource": [
                "arn:aws:s3:::your-bucket/*"
            ]
        }
    ]
 }

 

Adding Logstash S3 inputs to my stack

To send data from an Amazon S3 Bucket, we need to add the configuration details to your Logit account.

  • Sign in to your Dashboard
  • Click "Settings" on the Stack that you want your Amazon S3 Bucket to send data to
  • Select "Logstash Inputs" from the menu on the left hand side of the page
  • Under Add new S3 Input, click the "Configure S3" button
  • Select "Pull logs from an s3 bucket"
  • Complete the required form as shown below, all fields are required apart from the prefix field which is optional.
  • Click "Configure Input" to securely send the details

  • bucket name - This is the name of your Amazon S3 Bucket
  • access key id - This is your AWS Access Key Id
  • secret access key - This is your AWS Secret Access Key
  • region - The AWS Region (e.g. us-west-2)
  • prefix - Optional - If specified, the prefix of filenames in the bucket must match (not a regexp)
  • type - Identifier for filtering logstash logs and metrics sent by this method

All new S3 inputs need to be approved before they can be used to send logs and metrics. Logit will contact you to finalise the configuration setup (this should be actioned in less than 24 hours). If approved Logit will then use these details to set up your Amazon S3 Bucket as a Logstash input. All information is sent to Logit securely, Logit takes data security very seriously. Read more on Logit security.
 

How frequently does the data get sent to Logit?

The Logit platform polls the Amazon S3 Bucket once per minute for new data. Any new data is then identified and sent to your Logit stack.
 

Viewing the Amazon S3 Bucket inputs currently available on a Stack

Any Amazon S3 Bucket input, both active and available, can be viewed on your Logstash Inputs page under S3 Inputs. Any input still pending verfication will show as disabled. To get here from your Dashboard:

  • Sign in to your Account
  • Click "Settings" on the Stack that you want your Amazon S3 Bucket to send data to
  • Select "Logstash Inputs" from the menu on the left hand side of the page
  • Under S3 Inputs both active and inactive inputs are shown as below:

Managing your Amazon S3 Bucket inputs

If you want to edit one of your Amazon S3 Bucket input details or wish to remove an input altogether you currently need to contact support via live chat or our contact us page. Customer support will be happy to offer assistance and to make any changes that you require.
 

What's next

Did this answer your question?