Skip to main content

Streaming of Access Logs

IO River provides the ability to get CDN access logs in a unified format. This means logs from all CDNs will be streamed using the same format for easy processing.

To set up logs, follow these two steps:

  • Configure a log destination.
  • Define a Behavior to stream logs to this destination.

Creating a Log Destination​

To create an access-log destination:

  1. Navigate to your service.

  2. In the sidebar, select Logs, and then select Log Destinations.

  3. Click the Add Destination button.

  4. Follow the wizard steps:

    • Define Destination
      • Destination Name - Type a friendly name for the destination.
      • Destination Type - Select the type of destination:
        • AWS S3 bucket - provide the bucket name and region.
        • S3 Compatible - provide the bucket name, region, and endpoint.
        • Hydrolix - provide the endpoint, project, and table.
    • Set Credentials
      • For an S3 bucket destination, pick either Assume Role or Access Key as the credentials type, and fill in your bucket credentials.
      • For a Hydrolix destination, enter the username, password, and an authentication token.
    • Delivery Trigger
      • Select the parameters that control how often a log file is sent.
    • Define Content
      • Select which fields should be included in the log.
      • If you select Client IP, select whether to anonymize the IP address or not.
      • IF you select request or response headers, enter the header names.
    • Define File Structure
      • For an S3 bucket destination, you can change the object names IO River uploads to your bucket
        • The object name must include the {service-name} and {random} keywords.
        • The object name can include the timestamp in your favorite structure, to help you digest data in the order it is generated.
      • Select the Log File Format
        • JSON List – A plain json list.
        • JSON Object – A json object containing a records field.
        • Delimited Line JSON – Each line is a json.
        • CSV – A csv file.

  5. Click Create Destination.

Once created, the new destination will show up in the list of logs destinations.

Streaming Logs to Destination​

In this step you will set up log streaming to the configured destination by configuring a Behavior. You can either create a new behavior, or add an Action to an existing one.

To stream logs by creating a new behavior:

  1. Navigate to your service.
  2. Choose the Behaviors tab.
  3. Click the Add New Behavior button.
  4. Fill out the new behavior form:
    • Name - Enter the name of the new behavior
    • Path Pattern - Wildcard expression which identifies the paths to apply the behavior on.
    • Click the Add Action button.
    • In the Action Type dropdown, select Stream Logs.
    • Log Destination - Select the log destination previously created.
    • Sample Rate - Type the percentage of logs which should be sampled and sent to the destination.

Streaming Logs from CloudFront​

When using CloudFront as a CDN provider, log samples are retrieved via CloudFront Real-Time Logs and Kinesis. If your CloudFront account is from IO River, then we set up everything automatically. If you bring your own CloudFront account, you should Setup a CloudFront Kinesis Stream.
This setup needs to be performed once for each AWS account used with IO River.

Note​

Configuring streaming logs on a path pattern with a high traffic volume and a high sample rate can result in a high log volume being sent to your log destination.