In order to send data to your Logit.io stacks from an Amazon S3 Bucket, you must update permissions in the AWS (Amazon Web Service) IAM Policy being used. IAM (the Identity and Access Management) is a web service that helps users securely control access to their AWS resources. Amazon IAM controls who is authenticated and authorised to use your Amazon S3 Bucket.
Preparing your AWS permissions
The following permission should be applied:
s3:ListBucket - to check if the S3 Bucket exists and list objects in it.
s3:GetObject - to check object metadata and download objects from S3 Buckets.
Your permissions should appear as follows:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SidID",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-bucket/*"
]
}
]
}
Adding Logstash S3 inputs to my stack
To send data from an Amazon S3 Bucket, we need to add the configuration details to your Logit.io account.
Sign in to your Dashboard
Click "Settings" on the Stack that you want your Amazon S3 Bucket to send data to
Select "Logstash Inputs" from the menu on the left-hand side of the page
Under Add new S3 Input, click the "Configure S3" button
Select "Pull logs from an s3 bucket"
Complete the required form as shown below, all fields are required apart from the prefix field which is optional.
Click "Configure Input" to securely send the details
bucket name - This is the name of your Amazon S3 Bucket
access key id - This is your AWS Access Key Id
secret access key - This is your AWS Secret Access Key
region - The AWS Region (e.g. us-west-2)
prefix - Optional - If specified, the prefix of filenames in the bucket must match (not a regexp)
type - Identifier for filtering logstash logs and metrics sent by this method
All new S3 inputs need to be approved before they can be used to send logs and metrics. Logit.io will contact you to finalise the configuration setup (this should be actioned in less than 24 hours). If approved Logit.io will then use these details to set up your Amazon S3 Bucket as a Logstash input. All information is sent to Logit.io securely, Logit.io takes data security very seriously. Read more on Logit.io security.
How frequently does the data get sent to Logit.io?
The Logit.io platform polls the Amazon S3 Bucket once per minute for new data. Any new data is then identified and sent to your Logit.io stack.
Viewing the Amazon S3 Bucket inputs currently available on a Stack
Any Amazon S3 Bucket input, both active and available, can be viewed on your Logstash Inputs page under S3 Inputs. Any input still pending verification will show as disabled. To get here from your Dashboard:
Sign in to your Account
Click "Settings" on the Stack that you want your Amazon S3 Bucket to send data to
Select "Logstash Inputs" from the menu on the left-hand side of the page
Under S3 Inputs both active and inactive inputs are shown as below:
Managing your Amazon S3 Bucket inputs
If you want to edit one of your Amazon S3 Bucket input details or wish to remove an input altogether you currently need to contact support via live chat or our contact us page. Customer support will be happy to offer assistance and to make any changes that you require.
What's next
How can I archive my logs, data and metrics after my retention timeframe has ended