Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Configuring an Amazon AWS CloudTrail Log Source by using the Amazon Web Services Protocol

 

If you want to collect AWS CloudTrail logs from CloudWatch logs, configure a log source on the JSA Console so that Amazon AWS CloudTrail can communicate with JSA by using the Amazon Web Services protocol.

  1. If automatic updates are not enabled, download and install the most recent version of the following RPMs from the https://support.juniper.net/support/downloads/ onto your JSA Console.

    • Protocol Common

    • Amazon AWS REST API Protocol RPM

    • Amazon Web Services Protocol RPM

    • DSMCommon RPM

    • Amazon AWS CloudTrail DSM RPM

  2. Choose which method you will use to configure an Amazon AWS CloudTrail log source by using the Amazon Web Services protocol.

Configuring an Amazon AWS CloudTrail Log Source by using the Amazon Web Services Protocol and Kinesis Data Streams

If you want to collect AWS CloudTrail logs from Amazon Kinesis Data Streams, configure a log source on the JSA Console so that Amazon AWS CloudTrail can communicate with JSA by using the Amazon Web Services protocol.

  1. Follow the procedures in the AWS online documentation Sending Events to CloudWatch Logs to configure CloudTrail to deliver the logs in a log group of the AWS CloudWatch Logs.

  2. Create CloudWatch Logs destinations and a CloudWatch Logs subscription filter.

    For more information about CloudWatch Logs Destinations and Subscriptions, see Cross-Account Log Data Sharing with Subscription.

    1. Create a CloudWatch Logs destination that points to a destination Kinesis Data Stream.

      Only one CloudWatch Logs destination is required per region and the destination Kinesis Data Stream can be in any region.

    2. Create a CloudWatch Logs subscription filter with a blank filter pattern to subscribe the destination to the CloudWatch Logs log group and match all events.

      The subscription filter is now associated with a Cloud Watch Logs log group that contains AWS CloudTrail logs, and delivers those logs to a Kineses Data Stream.

  3. Adding an Amazon AWS CloudTrail log source by using the Amazon Web Services protocol and Kinesis Data Streams.

Adding an Amazon AWS CloudTrail log source by using the Amazon Web Services protocol and Kinesis Data Streams

If you want to collect AWS CloudTrail logs from Amazon Kinesis Data Streams, add a log source on the JSA Console so that Amazon AWS CloudTrail can communicate with JSA by using the Amazon Web Services protocol.

  1. Use the following table describes the parameters that require specific values to collect audit events from Amazon AWS CloudTrail by using the Amazon Web Services protocol:

    Table 1: Amazon Web Services log source parameters for Amazon Kinesis Data Streams

    Parameter

    Description

    Protocol Configuration

    Select Amazon Web Services from the Protocol Configuration list.

    Authentication Method

    Access Key ID / Secret Key

    Standard authentication that can be used from anywhere.

    Assume IAM Role

    Authenticate with keys and then temporarily assume a role for access.

    EC2 Instance IAM Role

    If your JSA managed host is running in an AWS EC2 instance, choosing this option uses the IAM role from the metadata that is assigned to the instance for authentication; no keys are required. This method works only for managed hosts that are running within an AWS EC2 container.

    Access Key

    The Access Key ID that was generated when you configured the security credentials for your AWS user account.

    If you selected Access Key ID / Secret Key or Assume IAM Role, the Access Key parameter displays.

    Secret Key

    The Secret Key that was generated when you configured the security credentials for your AWS user account.

    If you selected Access Key ID / Secret Key or Assume IAM Role, the Secret Key parameter displays.

    Assume Role ARN

    The full ARN of the role to assume. It must begin with "arn:" and can't contain any leading or trailing spaces, or spaces within the ARN.

    If you selected Assume IAM Role, the Assume Role ARN parameter displays.

    Assume Role Session Name

    The session name of the role to assume. The default is JSAAWSSession. Leave as the default if you don't need to change it. This can only contain upper-case and lower-case alphanumeric characters, underscores, or any of the following characters: =,.@-

    If you selected Assume IAM Role, the Assume Role Session Name parameter displays.

    Regions

    Toggle each region that is associated with the Amazon Web Service that you want to collect logs from.

    AWS Service

    From the AWS Service list, select Kinesis Data Streams.

    Kinesis Data Streams

    The Kinesis Data Stream from which to consume data.

    Enable Kinesis Advanced Options

    Enable the following optional advanced configuration values. Advanced options values are only used when this option is chosen, otherwise the default values are used.

    Initial Position in Stream

    This option controls which data to pull on a newly configured log source. Select Latest to pull the latest data that is available. Select Trim Horizon to pull the oldest data that is available.

    Kinesis Worker Thread Count

    The number of worker threads to use for Kinesis Data Stream processing. Each worker thread can process approximately 10000 - 20000 events per second depending on record size and system load. If your log source is not able to process the new data in the stream, you can increase the number of threads here to a maximum of 16. The allowed range is 1 - 16. The default value is 2.

    Checkpoint Interval

    The interval (in seconds) at which to checkpoint data sequence numbers. Each record from a shard in a Kinesis Data Stream has a sequence number. Checkpointing your position allows this shard to resume processing at the same point if processing fails or a service restarts. A more frequent interval reduces data duplication but increases Amazon Dynamo DB usage. The allowed range is 1 - 3600 seconds. The default is 10 seconds.

    Kinesis Application

    (Optional) Leave this option blank to have this log source consume data from all available shards in the Kinesis Data Stream. To have multiple log sources on multiple event processors scale log consumption without loss or duplication, use a common Kinesis Application across those log sources. (Example: ProdKinesisConsumers)

    Partition

    (Optional) Select this option to collect data from a specific partition in the Kinesis Data Stream by specifying a partition name here.

    Extract Original Event

    To forward only the original event that was added to the Kinesis Data Stream to JSA, select this option.

    Kinesis logs wrap the events that they receive with extra metadata. Select this option if you want only the original event that was sent to AWS without the additional stream metadata through Kinesis.

    The original event is the value for the message key that is extracted from the Kinesis log. The following Kinesis logs event example shows the original event that is extracted from the Kinesis log in highlighted text:

    {"owner":"123456789012","subscriptionFilters":["allEvents"], "logEvents": [{"id"

    :"35093963143971327215510178578576502306458824699048362100", "message":" {\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\": \"AssumedRole\",

    \"principalId\":\"ARO1GH58EM3ESYDW3XHP6:

    test_session\", \"arn\":\"arn:aws:sts::123456789012:assumed-role\/

    CVDevABRoleToBeAssumed\ /test_visibility_session\",\"accountId\":\"123456789012\",\"accessKeyId \"

    :\"ASIAXXXXXXXXXXXXXXXX\",\"sessionContext\":{\"sessionIssuer\":{\"type \"

    :\"Role\",\"principalId\":\"AROAXXXXXXXXXXXXXXXXX\",\"arn\":\ "arn:aws:iam::123456789012:role\/CVDevABRoleToBeAssumed\",\"accountId\" :\"123456789012\",\"userName\":\"CVDevABRoleToBeAssumed\"}, \"webIdFederationData\":{},\"attributes\":{\"mfaAuthenticated\" :\"false\",\"creationDate\":\"2019-11-13T17:01:54Z\"}}},\"eventTime\":\ "2019-11-13T17:43:18Z\",\"eventSource\":\"cloudtrail.amazonaws.com\", \"eventName\":\"DescribeTrails\",\"awsRegion\":\"ap-northeast-1\",\ "sourceIPAddress\":\"192.0.2.1\",\"requestParameters\":null, \"responseElements\":null,\"requestID\":\ "41e62e80-b15d-4e3f-9b7e-b309084dc092\",\"eventID\": \"904b3fda-8e48-46c0-a923-f1bb2b7a2f2a\",\"readOnly\":true,\"eventType \" :\"AwsApiCall\",\"recipientAccountId\":\"123456789012\"} ","timestamp": 1573667733143}],"messageType":"DATA_MESSAGE","logGroup":"CloudTrail\

    /DefaultLogGroup","logStream":"123456789012_

    CloudTrail_us-east-2_2"}

    Use As A Gateway Log Source

    If you do not want to define a custom log source identifier for events, clear the checkbox.

    If you don't select Use As A Gateway Log Source and you don't configure the Log Source Identifier Pattern, JSA receives events as unknown generic log sources.

    Log Source Identifier Pattern

    If you selected Use As A Gateway Log Source, use this option to define a custom log source identifier for events that are being processed and for log sources to be automatically discovered when applicable. If you don't configure the Log Source Identifier Pattern, JSA receives events as unknown generic log sources.

    Use key-value pairs to define the custom Log Source Identifier. The key is the Identifier Format String, which is the resulting source or origin value. The value is the associated regex pattern that is used to evaluate the current payload. This value also supports capture groups that can be used to further customize the key.

    Define multiple key-value pairs by typing each pattern on a new line. Multiple patterns are evaluated in the order that they are listed. When a match is found, a custom Log Source Identifier displays.

    The following examples show multiple key-value pair functions.

    Patterns

    VPC=\sREJECT\sFAILURE

    $1=\s(REJECT)\sOK

    VPC-$1-$2=\s(ACCEPT)\s(OK)

    Events

    {LogStreamName: LogStreamTest,Timestamp: 0,Message: ACCEPT OK,IngestionTime: 0,EventId: 0}

    Resulting custom log source identifier

    VPC-ACCEPT-OK

    Use Proxy

    If JSA accesses the Amazon Web Service by using a proxy, select this option.

    If the proxy requires authentication, configure the Proxy Server, Proxy Port, Proxy Username, and Proxy Password fields.

    If the proxy does not require authentication, configure the Proxy IP or Hostname field.

    EPS Throttle

    The upper limit for the maximum number of events per second (EPS). The default is 5000.

    If the Use As A Gateway Log Source option is selected, this value is optional.

    If the EPS Throttle parameter value is left blank, no EPS limit is imposed by JSA.

  2. To verify that JSA is configured correctly, review the following table to see an example of a parsed event message.

    The actual CloudTrail logs are wrapped in a Kinesis Data Streams JSON payload:

    Table 2: Kinesis Data Streams sample message supported by the Amazon AWS CloudTrail DSM

    Event name

    Low-level category

    Sample log message

    Describe Trails

    Read Activity Attempted

    {"owner":"123456789012","subscriptionFilters":["allEvents"],
    "logEvents":
    [{"id":"35101382794889527301913782399021634305485606205478862909",
    "message":"{\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\":\
    "AssumedRole\",\"principalId\":\"AROA3GFMEP3ESYDW3XHP6:
    cloud_visibility_session\",\"arn\":\"arn:aws:sts::123456789012:
    assumed-role\/CVDevABRoleToBeAssumed\/cloud_visibility_session\",
    \"accountId\":\"123456789012\",\"accessKeyId\":
    \"ASIA3ABCDE3E6ZUV7IF5\",\"sessionContext\":{\"sessionIssuer\":
    {\"type\":\"Role\",\"principalId\":\"AROA3GABCD3ESYDW3XHP6\",\"arn\
    ":\"arn:aws:iam::123456789012:role\/CVDevABRoleToBeAssumed\",
    \"accountId\":\"123456789012\"\"userName\":\
    "CVDevABRoleToBeAssumed\"},\"webIdFederationData\":{},\
    "attributes\":{\"mfaAuthenticated\":\"false\",\"creationDate\":\
    "2019-11-17T13:34:07Z\"}}},\"eventTime\":\"2019-11-17T14:10:48Z\",
    \"eventSource\":\"cloudtrail.amazonaws.com\",\"eventName\":\
    "DescribeTrails\",\"awsRegion\":\"ap-northeast-3\",
    \"sourceIPAddress\":\"192.0.2.1\",\"requestParameters\":null,
    \"responseElements\":null,\"requestID\":\
    "31afb5b7-6857-467a-bce4-835ee7d02ad2\",\"eventID\":
    \"26caf544-010c-423a-88a1-ca71cbd243ca\",\"readOnly\"
    :true,\"eventType\":\"AwsApiCall\",\"recipientAccountId\":\
    "123456789012\"}" ,"timestamp":1574000441797}],"messageType":
    "DATA_MESSAGE","logGroup":"CloudTrail\/DefaultLogGroup",
    "logStream":"123456789012_CloudTrail_us-east-2"}

Configuring an Amazon AWS CloudTrail Log Source by using the Amazon Web Services Protocol and CloudWatch Logs

If you want to collect AWS CloudTrail logs from Amazon CloudWatch logs, configure a log source on the JSA Console so that Amazon AWS CloudTrail can communicate with JSA by using the Amazon Web Services protocol.

  1. Creating an Identity and Access (IAM) user in the AWS Management Console when using Amazon Web Services
  2. Creating a log group in Amazon CloudWatch Logs to retrieve Amazon CloudTrail logs in JSA
  3. Configure Amazon AWS CloudTrail to send log files to CloudWatch Logs
  4. Configuring security credentials for your AWS user account
  5. Adding an Amazon AWS CloudTrail log source by using the Amazon Web Services protocol and CloudWatch Logs

Creating an Identity and Access (IAM) user in the AWS Management Console when using Amazon Web Services

Web Services An Amazon administrator must create a user and then apply the CloudWatchLogsReadOnlyAccess policy in the AWS Management Console. The JSA user can then create a log source in JSA.

  1. Create a user:
    1. Log in to the AWS Management Console as an administrator.

    2. Create an Amazon AWS IAM user and then apply the CloudWatchLogsReadOnlyAccess policy.

Creating a log group in Amazon CloudWatch Logs to retrieve Amazon CloudTrail logs in JSA

You must create a log group in Amazon CloudWatch Logs to make the CloudTrail log available for JSA polling.

  1. Log in to your CloudWatch console at this link: https://console.aws.amazon.com/cloudwatch.
  2. Select Logs from left navigation pane.
  3. Click Actions > Create Log Group
  4. Type the name of your Log Group. For example, CloudTrailAuditLogs.
  5. Click Create log group.

Configure Amazon AWS CloudTrail to send log files to CloudWatch Logs

You must configure CloudTrail to deliver the logs in a log group of the AWS CloudWatch Logs.

Configuring security credentials for your AWS user account

You must have your AWS user account access key and the secret access key values before you can configure a log source in JSA.

  1. Log in to your IAM console
  2. Select Users from left navigation pane and then select your user name from the list.
  3. Click the Security Credentials tab.
  4. In the Access Keys section, click Create access key.
  5. From the window that displays after the access key and corresponding secret access key are created, download the .csv file that contains the keys or copy and save the keys.Note

    Save the Access key ID and Secret access key and use them when you configure a log source in JSA.

    Note

    You can view the Secret access key only when it is created.

Adding an Amazon AWS CloudTrail log source by using the Amazon Web Services protocol and CloudWatch Logs

If you want to collect AWS CloudTrail logs from Amazon CloudWatch logs, add a log source on the JSA Console so that Amazon AWS CloudTrail can communicate with JSA by using the Amazon Web Services protocol.

  1. Use the following table describes the parameters that require specific values to collect audit events from Amazon AWS CloudTrail by using the Amazon Web Services protocol:

    Table 3: Amazon Web Services log source parameters for AWS CloudWatch Logs

    Parameter

    Description

    Protocol Configuration

    Select Amazon Web Services from the Protocol Configuration list.

    Authentication Method

    Access Key ID / Secret Key

    Standard authentication that can be used from anywhere.

    Assume IAM Role

    Authenticate with keys and then temporarily assume a role for access.

    EC2 Instance IAM Role

    If your JSA managed host is running in an AWS EC2 instance, choosing this option uses the IAM role from the metadata that is assigned to the instance for authentication; no keys are required. This method works only for managed hosts that are running within an AWS EC2 container.

    Access Key

    The Access Key ID that was generated when you configured the security credentials for your AWS user account.

    If you selected Access Key ID / Secret Key or Assume IAM Role, the Access Key parameter displays.

    Secret Key

    The Secret Key that was generated when you configured the security credentials for your AWS user account.

    If you selected Access Key ID / Secret Key or Assume IAM Role, the Secret Key parameter displays.

    Assume Role ARN

    The full ARN of the role to assume. It must begin with "arn:" and can't contain any leading or trailing spaces, or spaces within the ARN.

    If you selected Assume IAM Role, the Assume Role ARN parameter displays.

    Assume Role Session Name

    The session name of the role to assume. The default is JSAAWSSession. Leave as the default if you don't need to change it. This can only contain upper-case and lower-case alphanumeric characters, underscores, or any of the following characters: =,.@-

    If you selected Assume IAM Role, the Assume Role Session Name parameter displays.

    Regions

    Toggle each region that is associated with the Amazon Web Service that you want to collect logs from.

    AWS Service

    From the AWS Service list, select CloudWatch Logs.

    Log Group

    The name of the log group in Amazon CloudWatch where you want to collect logs from.

    Note: A single log source collects CloudWatch Logs from 1 log group at a time. If you want to collect logs from multiple log groups, create a separate log source for each log group.

    Enable CloudWatch Advanced Options

    Enable the following optional advanced configuration values. Advanced options values are only used when this option is chosen, otherwise the default values are used.

    Log Stream

    (Optional) The name of the log stream within a log group. If you want to collect logs from all log streams within a log group, leave this field blank.

    Filter Pattern

    (Optional) Type a pattern for filtering the collected events. This pattern is not a regex filter. Only the events that contain the exact value that you specified are collected from CloudWatch Logs. If you type ACCEPT as the Filter Pattern value, only the events that contain the word ACCEPT are collected, as shown in the following example.

    {LogStreamName: LogStreamTest,Timestamp: 0, Message: ACCEPT OK,IngestionTime: 0,EventId: 0}

    Event Delay

    Delay in seconds for collecting data.

    Other Region(s)

    Deprecated. Use Regions instead.

    Extract Original Event

    To forward only the original event that was added to the CloudWatch Logs to JSA, select this option.

    CloudWatch logs wrap the events that they receive with extra metadata. Select this option if you want to collect only the original event that was sent to AWS without the additional stream metadata through CloudWatch Logs.

    The original event is the value for the message key that is extracted from the CloudWatch log. The following CloudWatch Logs event example shows the original event that is extracted from CloudWatch Logs in highlighted text:

    {LogStreamName: 123456786_CloudTrail_us-east-2,Timestamp: 1505744407363, Message: {"eventVersion":"1.05","userIdentity": {"type":"IAMUser","principalId":"AAAABBBCCCDDDBBBCCC","arn": "arn:aws:iam::1234567890:user/<username>",accountId":"1234567890","accessKeyId" :"AAAABBBBCCCCDDDD","userName":"User-Name", "sessionContext":{"attributes":{"mfaAuthenticated": "false","creationDate":"2017-09-18T13:22:10Z"}}, "invokedBy":"signin.amazonaws.com"},"eventTime": "2017-09-18T14:10:15Z","eventSource": "cloudtrail.amazonaws.com","eventName": "DescribeTrails","awsRegion":"us-east-1", "sourceIPAddress":"192.0.2.1","userAgent": "signin.amazonaws.com","requestParameters": {"includeShadowTrails":false,"trailNameList": []},"responseElements":null,"requestID": "11b1a00-7a7a-11a1-1a11-44a4aaa1a","eventID": "a4914e00-1111-491d-bbbb-a0dd3845b302","eventType": "AwsApiCall","recipientAccountId":"1234567890"} , IngestionTime: 1505744407506, EventId: 335792223611111122479126672222222513333}

    Use As A Gateway Log Source

    If you do not want to define a custom log source identifier for events, clear the checkbox.

    If you don't select Use As A Gateway Log Source and you don't configure the Log Source Identifier Pattern, JSA receives events as unknown generic log sources.

    Log Source Identifier Pattern

    If you selected Use As A Gateway Log Source, use this option to define a custom log source identifier for events that are being processed and for log sources to be automatically discovered when applicable. If you don't configure the Log Source Identifier Pattern, JSA receives events as unknown generic log sources.

    Use key-value pairs to define the custom Log Source Identifier. The key is the Identifier Format String, which is the resulting source or origin value. The value is the associated regex pattern that is used to evaluate the current payload. This value also supports capture groups that can be used to further customize the key.

    Define multiple key-value pairs by typing each pattern on a new line. Multiple patterns are evaluated in the order that they are listed. When a match is found, a custom Log Source Identifier displays.

    The following examples show multiple key-value pair functions.

    Patterns

    VPC=\sREJECT\sFAILURE

    $1=\s(REJECT)\sOK

    VPC-$1-$2=\s(ACCEPT)\s(OK)

    Events

    {LogStreamName: LogStreamTest,Timestamp: 0,Message: ACCEPT OK,IngestionTime: 0,EventId: 0}

    Resulting custom log source identifier

    VPC-ACCEPT-OK

    Automatically Acquire Server Certificate(s)

    Select Yes for JSA to automatically download the server certificate and begin trusting the target server.

    You can use this option to initialize a newly created log source and obtain certificates, or to replace expired certificates.

    Use Proxy

    If JSA accesses the Amazon Web Service by using a proxy, select this option.

    If the proxy requires authentication, configure the Proxy Server, Proxy Port, Proxy Username, and Proxy Password fields.

    If the proxy does not require authentication, configure the Proxy IP or Hostname field.

    EPS Throttle

    The upper limit for the maximum number of events per second (EPS). The default is 5000.

    If the Use As A Gateway Log Source option is selected, this value is optional.

    If the EPS Throttle parameter value is left blank, no EPS limit is imposed by JSA.

  2. To verify that JSA is configured correctly, review the following table to see an example of a parsed event message.

    The actual CloudTrail logs are wrapped in a CloudWatch logs JSON payload:

    Table 4: Amazon CloudTrail Logs sample message supported by the Amazon AWS CloudTrail DSM

    Event name

    Low-level category

    Sample log message

    Describe Trails

    Read Activity Attempted

    {LogStreamName: 1234567890_CloudTrail_us
    -east-2,Timestamp: 1505744407363,Message:
    {"eventVersion":"1.05","userIdentity":{"type"
    :"IAMUser","principalId":"AIDAIEGANDWTHAAUMATYA",
    "arn":"arn:aws:iam::1234567890:user/QRadar-ITeam",
    "accountId":"1234567890","accessKeyId":
    "AAAABBBBCCCCDDDD","userName":"QRadar-ITeam",
    "sessionContext":{"attributes":{"mfaAuthenticated":
    "false","creationDate":"2017-09-18T13:22:10Z"}},
    "invokedBy":"signin.amazonaws.com"},"eventTime":
    "2017-09-18T14:10:15Z","eventSource":
    "cloudtrail.amazonaws.com","eventName":
    "DescribeTrails","awsRegion":"us-east-1",
    "sourceIPAddress":"127.0.0.1","userAgent":
    "signin.amazonaws.com","requestParameters":
    {"includeShadowTrails":false,"trailNameList":
    []},"responseElements":null,"requestID":
    "17b7a04c-99cca-11a1-9d83-43d5bce2d2fc",
    "eventID":"a4444e00-55e5-4444-bbbb-a0dd3845b302",
    "eventType":"AwsApiCall","recipientAccountId":
    "1234567890"} ,IngestionTime: 1505744407506,
    EventId: 33579222362711111111111111222222222222}