Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Create an SQS Queue and Configure S3 ObjectCreated Notifications

Before you can add a log source in JSA, you must create an SQS queue and configure S3 ObjectCreated notifications in the AWS Management Console when using the Amazon AWS S3 REST API protocol.

Complete the following procedures:

  1. Finding the S3 Bucket that contains the Data that you want to Collect
  2. Creating the SQS Queue that is used to Receive ObjectCreated Notifications
  3. Setting up SQS Queue Permissions
  4. Creating ObjectCreated Notifications

Finding the S3 Bucket that contains the Data that you want to Collect

You must find and note the region for S3 bucket that contains the data that you want to collect.

  1. Log in to the AWS Management Console as an administrator.
  2. Click Services, and then go to S3.
  3. From the AWS Region column in the Buckets list, note the region where the bucket that you want to collect data from is located. You need the region for the Region Name parameter value when you add a log source in JSA.
  4. Enable the check box beside the bucket name, and then from the panel that opens to the right, click Copy Bucket ARN to copy the value to the clipboard. Save this value or leave it on the clipboard. You need this value when you set up SQS queue permissions.

Creating the SQS Queue that is used to Receive ObjectCreated Notifications

You must create an SQS queue and configure S3 ObjectCreated notifications in the AWS Management Console when using the Amazon AWS REST API protocol.

You must complete Finding the S3 Bucket that contains the data that you want to collect. The SQS Queue must be in the same region as the AWS S3 bucket that the queue is collecting from.

  1. Log in to the AWS Management Console as an administrator.
  2. Click Services, and then go to the Simple Queue Service Management Console.
  3. In the upper right of the window, change the region to where the bucket is located. You noted this value when you completed the Finding the S3 Bucket that contains the data that you want to collect procedure.
  4. Select Create New Queue, and then type a value for the Queue Name.
  5. Click Standard Queue, select Configure Queue, and then change the default values for the following Queue Attributes.

    • Default Visibility Timeout - 60 seconds (You can use a lower value. In the case of load balanced collection, duplicate events might occur with values of less than 30 seconds. This value can't be 0.)

    • Message Retention Period - 14 days (You can use a lower value. In the event of an extended collection, data might be lost.)

    Use the default value for the remaining Queue Attributes.

    More options such as Redrive Policy or SSE can be used depending on the requirements for your AWS environment. These values should not affect the data collection.

  6. Select Create Queue.

Setting up SQS Queue Permissions

You must set up SQS queue permissions for users to access the queue.

You must complete Creating the SQS Queue that is used to Receive ObjectCreated Notifications.

You can set the SQS queue permissions by using either the Permissions Editor or a JSON policy document.

  1. Log in to the AWS Management Console as an administrator.
  2. Go to the SQS Management Console, and then select the queue that you created from the list.
  3. From the Properties window, select Details, and record the ARN field value.

    Example: arn:aws:sqs:us-east-1:123456789012:MySQSQueueName

  4. To set the SQS queue permissions by using the Permissions Editor, complete the following steps.
    1. From the Properties window, select Permissions > Add a Permission, and then configure the following parameters:
      Table 1: Permission Parameters

      Parameter

      Value

      Effect

      Click Allow.

      Principal

      Click Everybody (*).

      Actions

      From the list, select SendMessage
    2. Click Add Conditionals (Optional), and then configure the following parameters:
      Table 2: Add Conditionals (Optional) Parameters

      Parameter

      Value

      Qualifier

      None

      Condition

      ARNLike

      Key

      Type aws:SourceArn.

      Value

      The ARN of the S3 bucket from when you completed the Finding the S3 Bucket that contains the Data that you want to Collect procedure.

      Example: aws:s3:::my-example-s3bucket
    3. Click Add Condition > Add Permission.
  5. To set the SQS queue permissions by using a JSON Policy Document, complete the following steps.
    1. In the Properties window, select Edit Policy Document (Advanced).
    2. Copy and paste the following JSON policy into the Edit Policy Document window:

      Copy and paste might not preserve the white space in the JSON policy. The white space is required. If the white space is not preserved when you paste the JSON policy, paste it into a text editor and restore the white space. Then, copy and paste the JSON policy from your text editor into the Edit Policy Document window.

  6. Click Review Policy. Ensure that the data is correct, and then click Save Changes.

Creating ObjectCreated Notifications

Configure ObjectCreated notifications for the folders that you want to monitor in the bucket.

  1. Log in to the AWS Management Console as an administrator.
  2. Click Services, go to S3, and then select a bucket.
  3. Click the Properties tab, and in the Events pane, click Add notification. Configure the parameters for the new event.

    The following table shows an example of an ObjectCreated notification parameter configuration:

    Table 3: Example: New ObjectCreated Notification Parameter Configuration

    Parameter

    Value

    Name

    Type a name of your choosing.

    Events

    Select All object create events.

    Prefix

    AWSLogs/

    Tip:

    You can choose a prefix that contains the data that you want to find, depending on where the data is located and what data that you want to go to the queue. For example, AWSLogs/, CustomPrefix/AWSLogs/, AWSLogs/123456789012/.

    Suffix

    json.gz

    Send to

    SQS queue

    Tip:

    You can send the data from different folders to the same or different queues to suit your collection or JSA tenant needs. Choose one or more of the following methods:

    • Different folders that go to different queues

    • Different folders from different buckets that go to the same queue

    • Everything from a single bucket that goes to a single queue

    • Everything from multiple buckets that go to a single queue

    SQS

    The Queue Name from step 4 of Creating the SQS Queue that is used to Receive ObjectCreated Notifications.
    Figure 1: Example: Events Example: Events

    In the example in figure 1 of a parameter configuration, notifications are created for AWSLogs/ from the root of the bucket. When you use this configuration, All ObjectCreated events trigger a notification. If there are multiple accounts and regions in the bucket, everything gets processed. In this example, json.gz is used. This file type can change depending on the data that you are collecting. Depending on the content in your bucket, you can omit the extension or choose an extension that matches the data you are looking for in the folders where you have events set up.

    After approximately 5 minutes, the queue that contains data displays. In the Messages Available column, you can view the number of messages.

    Figure 2: Number of Available Messages Number of Available Messages
  4. Click Services, then go to Simple Queue Services.
  5. Right-click the Queue Name from step 4 of Creating the SQS Queue that is used to Receive ObjectCreated Notifications, then select View/Delete Messages to view the messages.
    Figure 3: SecureQueue TEST List SecureQueue TEST List

    Sample message:

  6. Click Services, then navigate to IAM.
  7. Set a User or Role permission to access the SQS queue and for permission to download from the target bucket. The user or user role must have permission to read and delete from the SQS queue. For information about adding, managing and changing permissions for IAM users, see the IAM Users documentation. After JSA reads the notification, and then downloads and processes the target file, the message must be deleted from the queue.

    Sample Policy:

    You can add multiple buckets to the S3 queue. To ensure that all objects are accessed, you must have a trailing /* at the end of the folder path that you added.

    You can add this policy directly to a user, a user role, or you can create a minimal access user with sts:AssumeRole permissions only. When you configure a log source in JSA, configure the assume Role ARN parameter for JSA to assume the role. To ensure that all files waiting to be processed in a single run (emptying the queue) can finish without retries, use the default value of 1 hour for the API Session Duration parameter.

    When you use assumed roles, ensure that the ARN of the user that is assuming the rule is in the Trusted Entities for that role. From the Trusted entities pane, you can view the trusted entities that can assume the role. In addition, the user must have permission to assume roles in that (or any) account.

    The following image example shows a sample Amazon AWS CloudTrail log source configuration in JSA.

    Tip:

    Use the Amazon AWS S3 REST API log source parameter values for your DSM when you configure your log source.
    Figure 4: Example: Amazon AWS CloudTrail log source configuration in JSA Example: Amazon AWS CloudTrail log source configuration in JSA