Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Configuring an Amazon AWS Route 53 Log Source by using an S3 Bucket with a Directory Prefix

You can collect AWS Route 53 Resolver query logs from a single account and region in an Amazon S3 bucket. Add a log source on the JSA Console so that Amazon AWS Route 53 can communicate with JSA by using the Amazon AWS S3 REST API protocol with a directory prefix.

Before you begin

If you have log sources in an S3 bucket from multiple regions or you are using multiple accounts, use the Configuring an Amazon AWS Route 53 log source that uses an S3 bucket with an SQS queue procedure.

A log source that uses directory prefix can retrieve data from only one region and one account. Use a different log source for each region and account. Include the region folder name in the file path for the Directory Prefix parameter value when you configure the log source.

  1. Configure Resolver query logging. When you configure the Query logs destination parameter, select S3 bucket for the value.
  2. Find an S3 bucket name and directory prefix for Amazon AWS Route 53.
  3. Create an Amazon AWS Identity and Access Management (IAM) user and then apply the AmazonS3ReadOnlyAccess policy.
  4. Configure the security credentials for you AWS user account.
  5. Amazon AWS S3 REST API log source parameters for Amazon AWS Route 53 when using a directory prefix.

Configuring Resolver Query Logging

Before you can add a log source in JSA, you must configure Resolver query logging on the AWS Management console.

  1. Log in to your AWS Management console to open the Route 53 console.
  2. From the Route 53 navigation menu, select Resolver > Query logging.
  3. From the region list, select the region where you want to create the query logging configuration.
    Tip:

    The region that you select must be the same region where you created the Amazon Virtual Private Clouds (VPCs) that you want to log queries for. If your VPCs are in multiple regions, create at least one query logging configuration for each region.
  4. Click Configure query logging, then type a name for your query logging configuration. Your configuration name displays in the console in the list of query logging configurations.
  5. In the Query logs destination section, select a destination where you want Resolver to publish query logs. JSA supports CloudWatch Logs log group and S3 bucket as destinations for query logs.

    • If you are using the Amazon AWS S3 REST API, select S3 bucket.

    • If you are using the Amazon Web Services protocol, select CloudWatch Logs log group.

  6. To log VPCs, in the VPCs to log queries for section, click Add VPC. DNS queries that originate in the VPCs that you select are logged. If you don't select any VPCs, no queries are logged by Resolver.
  7. Click Configure query logging.

Finding an S3 Bucket Name and Directory Prefix

Before you can add a log source in JSA, an Amazon administrator must create a user and then apply the AmazonS3ReadOnlyAccess policy in the AWS Management Console.

Before you begin

Alternatively, you can assign more granular permissions to the bucket. The minimum required permissions are s3:listBucket and s3:getObject.

For more information about permissions that are related to bucket operations, see the AWS documentation.

  1. Log in to the AWS Management Console as Administrator.
  2. Click Services.
  3. From the list, select Route 53.
  4. From the Route 53 navigation menu, select Query Logging.
  5. Note the S3 bucket name in the Destination ARN field. You need this value when you configure a log source in JSA. If the location path for the S3 Bucket name is available, note it as well.

Creating an Identity and Access Management (IAM) user in the AWS Management Console

An Amazon administrator must create a user and then apply the s3:listBucket and s3:getObject permissions to that user in the AWS Management Console. The JSA user can then create a log source in JSA.

The minimum required permissions are s3:listBucket and s3:getObject. You can assign other permissions to the user as needed.

Sample policy:

For more information about permissions that are related to bucket operations, go to the AWS documentation website.

  1. Log in to the AWS Management Console as an administrator.
  2. Click Services.
  3. From the list, select IAM.
  4. Click Users > Add user.
  5. Create an Amazon AWS IAM user and then apply the AmazonS3ReadOnlyAccess policy.

Configuring Security Credentials for your AWS User Account

You must have your AWS user account access key and the secret access key values before you can configure a log source in JSA.

  1. Log in to your IAM console.
  2. Select Users from left navigation pane and then select your user name from the list.
  3. To create the access keys, click the Security Credentials tab, and in the Access Keys section, click Create access key.
  4. Download the CSV file that contains the keys or copy and save the keys.
    Tip:

    Save the Access key ID and Secret access key. You need them when you configure a log source in JSA.

    You can view the Secret access key only when it is created.

    Add a log source on the JSA Console using a directory prefix.

Amazon AWS S3 REST API Log Source Parameters for Amazon AWS Route 53 when using a Directory Prefix

If you want to collect AWS Route 53 Resolver query logs from a single account and region in an Amazon S3 bucket, add a log source on the JSA Console that uses the Amazon AWS S3 REST API protocol with a directory prefix.

When you use the Amazon AWS S3 REST API protocol with a directory prefix, there are specific parameters that you must configure.

The following table describes the parameters that require specific values to collect Amazon AWS S3 REST API events from Amazon AWS Route 53:

Table 1: Amazon AWS S3 REST API log source parameters for the Amazon AWS Route 53 DSM

Parameter

Value

Log Source type

Amazon AWS Route 53

Protocol Configuration

 Amazon AWS S3 REST API

Log Source Identifier

Type a unique name for the log source.

The Log Source Identifier can be any valid value and does not need to reference a specific server. The Log Source Identifier can be the same value as the Log Source Name. If you have more than one Amazon AWS Route 53 log source that is configured, you might want to identify the first log source as awsroute53-1, the second log source as awsroute53-2, and the third log source as awsroute53-3.

Authentication Method

Access Key ID / Secret Key

Standard authentication that can be used from anywhere.

For more information about configuring security credentials, see Configuring security credentials for your AWS user account.

Assume IAM Role

Authenticate with keys and then temporarily assume a role for access. This option is available only when you use the SQS Event Notifications collection method.

For more information about creating IAM users and assigning roles, see Creating an Identity and Access Management (IAM) user in the AWS Management Console.

EC2 Instance IAM Role

If your managed host is running on an AWS EC2 instance, choosing this option uses the IAM Role from the instance metadata that is assigned to the instance for authentication; no keys are required. This method works only for managed hosts that are running within an AWS EC2 container.

Access Key ID

If you selected Access Key ID / Secret Key for the Authentication Method, the Access Key ID parameter is displayed.

The Access Key ID that was generated when you configured the security credentials for your AWS user account. This value is also the Access Key ID that is used to access the AWS S3 bucket.

Secret Key

If you selected Access Key ID / Secret Key for the Authentication Method, the Secret Key ID parameter is displayed.

The Secret Key that was generated when you configured the security credentials for your AWS user account. This value is also the Decret Key ID that is used to access the AWS S3 bucket.

Event Format

 Select AWS Cloud Trail JSON. The log source retrieves JSON formatted events.

S3 Collection Method

 Select Use a Specific Prefix.

Bucket Name

 The name of the AWS S3 bucket where the log files are stored.

Directory Prefix

The root directory location on the AWS S3 bucket from where the Resolver logs are retrieved; for example, AWSLogs/<AccountNumber>/Resolver/<RegionName>/

To pull files from the root directory of a bucket, you must use a forward slash (/) in the Directory Prefix file path.

Note:

  • Changing the Directory Prefix value clears the persisted file marker. All files that match the new prefix are downloaded in the next pull.

  • The Directory Prefix file path cannot begin with a forward slash (/) unless only the forward slash is used to collect data from the root of the bucket.

  • If the Directory Prefix file path is used to specify folders, you must not begin the file path with a forward slash (for example, use folder1/folder2 instead).

Region Name

The region that the SQS Queue or the AWS S3 bucket is in.

Example: us-east-1, eu-west-1, ap-northeast-3

Use as a Gateway Log Source

 Select this option for the collected events to flow through the JSA Traffic Analysis engine and for JSA to automatically detect one or more log sources.

Log Source Identifier Pattern

This option is available when Use as a Gateway Log Source is set to yes.

Use this option if you want to define a custom Log Source Identifier for events being processed. This field accepts key value pairs to define the custom Log Source Identifier, where the key is the Identifier Format String, and the value is the associated regex pattern. You can define multiple key value pairs by entering a pattern on a new line. When multiple patterns are used, they are evaluated in order until a match is found and a custom Log Source Identifier can be returned.

Show Advanced Options

 Select this option if you want to customize the event data.

File Pattern

This option is available when you set Show Advanced Options to Yes.

Type a regex for the file pattern that matches the files that you want to pull; for example, .*? \.json\.gz

Local Directory

This option is available when you set Show Advanced Options to Yes.

The local directory on the Target Event Collector. The directory must exist before the AWS S3 REST API PROTOCOL attempts to retrieve events.

S3 Endpoint URL

This option is available when you set Show Advanced Options to Yes.

The endpoint URL that is used to query the AWS S3 REST API.

If your endpoint URL is different from the default, type your endpoint URL. The default is https://s3.amazonaws.com.

Use S3 Path-Style Access

Forces S3 requests to use path-style access.

This method is deprecated by AWS. However, it might be required when you use other S3 compatible APIs. For example, the https://s3.region.amazonaws.com/bucket-name/key-name path-style is automatically used when a bucket name contains a period (.). Therefore, this option is not required, but can be used.

Use Proxy

If JSA accesses the Amazon Web Service by using a proxy, enable Use Proxy.

If the proxy requires authentication, configure the Proxy Server, Proxy Port, Proxy Username, and Proxy Password fields.

If the proxy does not require authentication, configure the Proxy IP or Hostname field.

Recurrence

How often a poll is made to scan for new data.

If you are using the SQS event collection method, SQS Event Notifications can have a minimum value of 10 (seconds). Because SQS Queue polling can occur more often, a lower value can be used.

If you are using the Directory Prefix event collection method, Use a Specific Prefix has a minimum value of 60 (seconds) or 1M. Because every listBucket request to an AWS S3 bucket incurs a cost to the account that owns the bucket, a smaller recurrence value increases the cost.

Type a time interval to determine how frequently the poll is made for new data. The time interval can include values in hours (H), minutes (M), or days (D). For example, 2H = 2 hours, 15M = 15 minutes, 30 = seconds.

EPS Throttle

The maximum number of events per second that are sent to the flow pipeline. The default is 5000.

Ensure that the EPS Throttle value is higher than the incoming rate or data processing might fall behind.

For more information about the Amazon AWS S3 REST API protocol, see Amazon AWS S3 REST API Protocol Configuration Options.