Basic Auditing on AWS

Prerequisites to Auditing

Before we dig into any of the specifics of auditing your account, you probably want to look at a few items first:

  • Enable Cloudtrail.
  • Enable Config.
  • Ensure logs are being generated and stored in S3 for both services.
  • Ensure versioning is enabled for all buckets with logs.
  • Ensure that life cycle rules aren't removing logs before the end of the desired audit period.
  • Ensure that only the root account can delete logs.

Auditing User Permissions in AWS IAM

AWS provides a very in-depth and mature mechanism to control user and application access in the AWS environment called IAM. When you first create an AWS account the best practice is to use the root account to configure IAM user access, then heavily secure your root user and even alert on it's usage. What we're going to look at in this section includes:

  1. Ensuring the root user access is secure, including alerting on it's usage.
  2. Using the IAM console to verify users and identify fields we want to review.
  3. Using the IAM policy simulator to test access to features we want restricted.
  4. Automating our review with the AWS CLI (or alternatively the SDK).

Securing your root user is a complex topic but essentially it boils down to:

  • Don't routinely use your root user.
  • Assign a physical MFA to the root user account. The physical token should not be difficult to retrieve if it's needed but shouldn't be immediately available to someone with the root credentials.
  • Monitor and alert on root account usage using CloudWatch Logs based alerting with CloudTrail.
  • Send emails with details of the actions carried out by the root account using CloudWatch Logs with Lambda.

We'll address how to do some of these things below, but otherwise they're relatively easy to implement. You can monitor and alert using CloudWatch Logs on CloudTrail, and you can set up events on your CloudTrail S3 bucket to kick off custom Lambda tasks.

To secure the root user:

  1. Note, To manage MFA devices for the root user account, you must be signed in as your root user. You cannot manage MFA devices for the root account using an IAM user. Keep in mind, your root user is essentially your god-mode.
  2. Log in.
  3. On the right side of the navigation bar, choose your account name, and choose Security Credentials.
  4. If necessary, choose Continue to Security Credentials.
  5. Then expand the Multi-Factor Authentication (MFA) section on the page.
  6. Click on the MFA option and follow the wizard.
  7. When done, click finish.

Next let's use the IAM console to verify a user:

  1. Open the IAM console.
  2. Click "Users".
  3. Click on a user.
  4. Review the different fields and tabs available to you.
  5. Pay special attention to the IAM permissions granted to the user.

Some fields that you might be interested in may include:

  • Username
  • Amazon Resource Name (ARN)
  • Has password (effectively has console access)
  • Groups, and the permissions of those groups
  • Creation date
  • Access keys and associated meta data
  • MFA devices

It's important to note that you should be auditing IAM Users, IAM Groups, and IAM Roles.

But now for my favourite tool: the IAM policy simulator. While we can do this in the console this is way easier to do on the command line:

aws iam simulate-principal-policy --policy-source-arn justin.fox --action-names "cloudtrail:DeleteTrail" "ec2:AuthorizeSecurityGroupIngress"

This simple snippet will test to see if my user has permission to the actions listed. It will return an allow or deny response based on whether it's allowed, implicitly denied, or explicitly denied. Typically if you haven't granted a permission it will show up as an "implicitDeny". If a deny statement is present it will show up as "explicitDeny". Typically you'll only see an explicit deny statement if you've set up IAM to be overly permissive and decided to only disallow Administrators some access. For example, you might have a group of super users with Administrator access but not CloudTrail access.

You can automate your review by iterating over your IAM users:

echo "ARN,FriendlyName,Type,Memberships,iam:CreateUser"
iamusers=`aws iam list-users --profile ${profile} | jq -c '.Users[]'`
for iamuser in $iamusers
do
  arn=`echo ${iamuser} | jq -c '.Arn' | tr -d '"'`
  friendlyname=`echo ${iamuser} | jq -c '.UserName' | tr -d '"'`
  groupnames=`aws iam list-groups-for-user --user-name ${friendlyname} --profile ${profile} | jq -c '.Groups[].GroupName' | tr '"' ' '| tr -d '\n'`
  CreateUser=`aws iam simulate-principal-policy --policy-source-arn ${arn} --action-names "iam:CreateUser" --profile ${profile} | jq -c '.EvaluationResults[].EvalDecision' | tr -d '"' | tr -d '\n'`
  echo "$arn,$friendlyname,user,$groupnames,$CreateUser"
done

Here's our completed example, hosted on github.

Auditing Security Groups in AWS EC2

In a similar vein to AWS IAM, AWS provides a fairly robust way to implement a local virtual firewall around your instances called Security Groups in the EC2 and VPC consoles. What we're going to look at in this section includes:

  1. Using the EC2 and VPC consoles to verify security groups and identify fields we want to review.
  2. Using Config to verify changes to a security group.
  3. Automating our review with the AWS CLI (or alternatively the SDK).

To start, lets use the EC2 console to verify security groups:

  1. Open the EC2 console.
  2. Under Network & Security, click "Security Groups".
  3. You'll see a list of security groups. Click on one.
  4. Review the allowed inbound and outbound traffic as well as tags.

Since the EC2 console tends to have a barrage of detail and while we can filter by VPC identifiers I tend to like the view from the VPC console:

  1. Open the VPC console.
  2. Under Security, click "Security Groups".
  3. By default you'll see the same list as the EC2 console. Let's narrow our focus.
  4. In the top left you'll see an option to filter by VPC.
  5. Select a VPC.
  6. You'll see an updated list of security groups. Click on one.
  7. Review the allowed inbound and outbound traffic as well as tags.
  8. For the next section, grab the security group identifier.

Now that we've captured a security group identifier that we want to review, we can review it in Config:

  1. Open the Config console.
  2. By default you'll land on the Config rules page, where you'll see rule statuses.
  3. Switch to the Resources page.
  4. You'll notice that you can search by resource identifier or by tag value. We're going to do a resource based search.
  5. Select the SecurityGroup resource type.
  6. Enter our noted security group identifier.
  7. Optional: if we were searching for a possibly deleted resource, check the "include deleted resources" option.
  8. Click "Look up".
  9. The resource should appear. If you have any rules, you'll see if the resource is compliant.
  10. Click on the icon under the config timeline column to see a visual timeline of the history of this resource.
  11. In this view you can traverse through time and see events that apply to this resource. You can see important information like what was changed (configuration/relationships) and you can see the relationship the resource had with other AWS resources.

Now let's practice making a point in time snapshot of a group of items we want available for a security group review. Ideally this is going to be in CSV format and done programmatically.

First we need to consider fields we want to include in our audit:

  • Security Group ID
  • Security Group Name
  • Direction (Ingress/Egress)
  • Protocol
  • Ports
  • CIDR/Destination

We can get almost all this information we want with a single AWS CLI command aws ec2 describe-security-groups.

securitygroups=`aws ec2 describe-security-groups --profile $profile --region $region | jq -c '.SecurityGroups[].GroupId' | tr -d '"' | tr -d ' '`
for securitygroup in $securitygroups
do
  secname=`aws ec2 describe-security-groups --group-ids ${securitygroup} --profile $profile --region $region | jq -c '.SecurityGroups[].GroupName' | tr -d '"' | tr -d ' '`
  vpcid=`aws ec2 describe-security-groups --group-ids ${securitygroup} --profile $profile --region $region | jq -c '.SecurityGroups[].VpcId' | tr -d '"' | tr -d ' '`
  sectype=""
  if [ "null" != "${vpcid}" ]
  then
    vpcname=`aws ec2 describe-tags --filters "Name=resource-id,Values=${vpcid}" "Name=key,Values=Name" --profile $profile --region $region | jq -c '.Tags[].Value' | tr -d '"' | tr -d ' '`
    sectype="${vpcname} (${vpcid})"
  else
    sectype="Classic"
  fi
  rulesin=`aws ec2 describe-security-groups --group-ids ${securitygroup} --profile $profile --region $region | jq -c '.SecurityGroups[].IpPermissions[]'`
  for rule in $rulesin
  do
    protocol=`echo "$rule" | jq -c '.IpProtocol' | tr -d '"' | tr -d ' '`
    fromport=`echo "$rule" | jq -c '.FromPort' | tr -d '"' | tr -d ' '`
    toport=`echo "$rule" | jq -c '.ToPort' | tr -d '"' | tr -d ' '`
    ipranges=`echo "$rule" | jq -c '.IpRanges[].CidrIp' | tr -d '"' | tr -d ' '`
    refsgids=`echo "$rule" | jq -c '.UserIdGroupPairs[].GroupId' | tr -d '"' | tr -d ' '`
    if [ "" != "${ipranges}" ]
    then
      for cidr in $ipranges
      do
        echo "$securitygroup,$secname,$sectype,Ingress,#,$protocol,$fromport,$toport,$cidr"
      done
    fi
    if [ "" != "${refsgids}" ]
    then
      for sgid in $refsgids
      do
        sgname=`aws ec2 describe-security-groups --group-ids ${sgid} --profile $profile --region $region 2> /dev/null | jq -c '.SecurityGroups[].GroupName' | tr -d '"' | tr -d ' '`
        if [ "" != "${sgname}" ]
        then
          echo "$securitygroup,$secname,$sectype,Ingress,#,$protocol,$fromport,$toport,$sgname ($sgid)"
        else
          echo "$securitygroup,$secname,$sectype,Ingress,#,$protocol,$fromport,$toport,$sgid (Name Error)"
        fi
      done
    fi
  done
done

Here's our completed example, hosted on github.

I think qwiklabs has a great example on how to do a similar example with python, here.

Auditing ACLs in AWS VPC

In the previous section we went over security groups which acted as host firewalls. In this section we'll address VPC ACLs which act as a network segment firewall. What we're going to look at in this section includes:

  1. Using the VPC console to verify ACLs and identify the fields we want to review.
  2. Using Config to verify changes to an ACL.
  3. Automating our review with the AWS CLI (or alternatively the SDK).

First up, let's get familiar with doing things in the VPC console:

  1. Open the VPC console.
  2. If you have multiple VPCs it will probably be easiest if you utilize the "filter by VPC" toggle in the top left.
  3. Under the security heading on the left, click "Network ACLs".
  4. You'll see a list of network ACLs. This is where tagging resources with names can become extremely useful.
  5. Select a network ACL. Note the network ACL id. Example, "acl-40a8d525".
  6. You'll have a few tabs available. Take time to look through each for details you might want included for an audit.
  7. An important detail might be which subnets an ACL is associated with.
  8. Another important detail might be inbound and outbound rules. Note that ACLs are stateless, unlike security groups. They're also evaluated in order.

Remember that network ACL id we noted earlier? Let's verify changes to it in Config:

  1. Open the Config console.
  2. By default you'll land on the Config rules page, where you'll see rule statuses.
  3. Switch to the Resources page.
  4. You'll notice that you can search by resource identifier or by tag value. We're going to do a resource based search.
  5. Select the NetworkACL resource type.
  6. Enter our noted acl-id.
  7. Optional: if we were searching for a possibly deleted resource, check the "include deleted resources" option.
  8. Click "Look up".
  9. The resource should appear. If you have any rules, you'll see if the resource is compliant.
  10. Click on the icon under the config timeline column to see a visual timeline of the history of this resource.
  11. In this view you can traverse through time and see events that apply to this resource. You can see important information like what was changed (configuration/relationships) and you can see the relationship the resource had with other AWS resources.

Since we've previously covered a very in-depth example of security groups, I'm going to skip the automation example here. But keep in mind that you should be able to achieve similar results!

Auditing API calls with AWS CloudTrail

CloudTrail records all API calls made to AWS. Since we enabled CloudTrail as part of our prerequisites section we should already have logged some events that should be viewable in the AWS CloudTrail console.

To review the processed CloudTrail logs in the console:

  1. Open the CloudTrail console.
  2. By default you should land on the API activity history page.
  3. By default you get a list of only API activity for create, modify, and delete API calls. If you need read calls, you'll need to access the raw data.
  4. You can play with the filter, values, and time ranges as desired.

Note that the CloudTrail console does not show read API calls. If you want to review read calls as well you'll need to download the raw S3 log files and process them.

Currently you can only view the last 7 days of logs via the console. If you need more than that you'll need to retrieve the raw data from S3. You can also log to CloudWatch Logs and set the retention period as desired. The raw json is available in CloudWatch logs if desired.

To review the raw CloudTrail logs:

  1. Open the CloudTrail console.
  2. Locate the active cloudtrail trail.
  3. Review your trail's settings, making note of the S3 bucket used.
  4. Go to the S3 console.
  5. Access the previously noted S3 bucket.
  6. Click into the AWSLogs folder.
  7. Traverse the folders into a date folder.
  8. You'll see json formatted gunzip'd data. Download any files you'd like to process.

Because the logs get written to S3, you can fire off an S3 event whenever a log file appears and have something like Lambda or SQS triggered. I find the Lambda trigger particularly useful to parse the log for events and notify an SNS topic as required or take other actions. Since CloudTrail can also integrate with CloudWatch Logs you can also define log metrics and alert filters for your API calls.

If you want to work with the CloudWatch Logs component:

  1. Open the CloudWatch console.
  2. Navigate to the CloudWatch Logs view (left hand link).
  3. Select your CloudTrail log group.
  4. Select your log stream.
  5. By default you'll see a bunch of json formatted events here.

I find that the CloudWatch logs interface for CloudTrail logs isn't the greatest, but it's still possible to run successful queries. It becomes more powerful if you create a metric filter on top of a log query and optionally create an alarm based on the metric filter.

For example, you could alert of root logins, cloudtrail disable/enable, large number of api calls from a user, and other such indicators.

To create a metric filter on your CloudTrail logs in CloudWatch Logs:

  1. Open the CloudWatch console.
  2. Navigate to the CloudWatch Logs view (left hand link).
  3. In the metric filters column, click the filters link for your CloudTrail logs group.
  4. Click "Add Metric Filter".
  5. Specify a filter pattern. Here's an example pattern for VPC ACL entries:
    { ($.eventName = "CreateNetworkAcl") || ($.eventName = "CreateNetworkAclEntry") || ($.eventName = "DeleteNetworkAcl") || ($.eventName = "DeleteNetworkAclEntry") || ($.eventName = "ReplaceNetworkAclEntry") || ($.eventName = "ReplaceNetworkAclAssociation") }
  6. Click test pattern to view the results against a log subset.
  7. Give the filter a human readable name. Continuing from our previous example, "NetworkACLEvents"
  8. Select the metric namespace or define a custom one.
  9. Provide a metric name (I usually use the filter name).
  10. Click "Create Filter".

Optionally, add an alarm to the metric filter:

  1. Open the CloudWatch console.
  2. Navigate to the CloudWatch Logs view (left hand link).
  3. In the metric filters column, click the filters link for your CloudTrail logs group.
  4. You'll see the list of created metric filters you've created.
  5. Click "Create Alarm".
  6. Provide a name and description.
  7. Specify the condition for when to alert.
  8. Define the actions for the different states. You can add notifications (SNS), auto scaling actions, and EC2 actions.
  9. Click "Create Alarm".

As a final note, if you're working with multiple accounts you'll want to aggregate all the log files to one of the accounts S3 buckets.

Auditing environment changes with AWS Config

Config records all changes made to AWS resources. Since we enabled Config as part of our prerequisites section we should already have logged some events that should be viewable in the AWS Config console.

To review the processed changes to resources in the console:

  1. Open the Config console.
  2. By default you should land on the resource inventory page.
  3. You can play with the filter, values, and whether or not you include deleted resources as desired.

Here's a quick visual example:

In case it's not obvious, clicking the small clock icon opens a visual timeline of the changes to that resource.

While you can have a SNS topic delivered for every change I usually find that generates a lot of unnecessary noise. Instead, building a filter using S3 events and Lambda functions to select the event conditions we need alerts for is a cleaner way to implement. This is particularly useful for custom alerting around security group changes. Using CloudTrail based alerting you can alert on the API call being used, but with Config you have details on items like ports, protocols, direction... etc.

To review the raw Config logs in S3:

  1. Open the Config console.
  2. Locate the Config setting.
  3. Review your Config's settings, making note of the S3 bucket used.
  4. Go to the S3 console.
  5. Access the previously noted S3 bucket.
  6. Click into the AWSLogs folder.
  7. Traverse the folders into a date folder.
  8. You'll see json formatted gunzip'd data. Download any files you'd like to process.

Because the logs get written to S3, you can fire off an S3 event whenever a log file appears and have something like Lambda or SQS triggered. I find the Lambda trigger particularly useful to parse the log for events and notify an SNS topic as required or take other actions. I prefer SNS to SES since SNS is easily subscribed to. You might prefer SES if you have a security distribution group you want emailed.

One very powerful feature of Config when it comes to auditing is Config Rules. You can select from a variety of AWS Managed Rules (such as the "cloudtrail-enabled" rule) or you can define a custom rule. Note that rules cost $2 each. If you're actually going for a compliance certificate that's probably more than reasonable, but for personal testing you might not want to forget that you enabled a rule.

If you're going for compliance, enabling and configuring all of these rules is probably a good place to start. But you're probably going to want to do more than just these basics - enter custom rules! With custom rules you can run either periodic checks against your rule or against configuration changes. However you choose to trigger your rule, you can then execute a lambda function. There are example lambda functions to get you started, but you can set a compliance rule for pretty much anything.

An example of a good custom rule might be verifying that no IAM roles have the managed Administrator policy attached. Now you could do this either as a rule (which shows up in a nice compliance dashboard and optionally sends an email or SNS) or just a lambda function that's triggered on S3 events to your Config log bucket.

Auditing Security with Trusted Advisor

Personally I only make use of the 4 free best practice checks offered by the AWS Trusted Advisor. The four free checks are:

  1. Service Limits
  2. IAM Use
  3. MFA on Root Account
  4. Security Groups (Specific Ports Unrestricted)

All four of these are super easy to pass and have implemented. If you're not passing all four, I suggest that you pause and think about what you're doing. Typically people find the free security group one the hardest for a green check mark - the only unrestricted ports you can have for a green check mark are ports 80, 25, 443, or 465. Certain sensitive ports are immediately flagged and get a red indicator, others merit only a yellow indicator. Typically you'll see a yellow indicator because someone's left SSH unrestricted. If you have a special use case, you can make a special exclusion for a security group.

If you're going for a compliance certification, you should definitely upgrade your support plan to get access to all of the trusted advisor checks as well as proper support.

Here are some of the checks that I have found particularly helpful:

  • AWS CloudTrail Logging
  • Exposed Access Keys
  • CloudFront SSL Certificate on the Origin Server
  • CloudFront Custom SSL Certificates in the IAM Certificate Store
  • CloudFront Alternate Domain Names
  • CloudFront Content Delivery Optimization
  • ELB Security Groups
  • ELB Listener Security
  • ELB Cross-Zone Load Balancing
  • ELB Connection Draining
  • Load Balancer Optimization
  • Amazon S3 Bucket Logging
  • Amazon Route 53 MX Resource Record Sets and Sender Policy Framework
  • Amazon Route 53 High TTL Resource Record Sets
  • Amazon Route 53 Name Server Delegations
  • Amazon Route 53 Alias Resource Record Sets
  • Amazon Route 53 Latency Resource Record Sets
  • Amazon EC2 Reserved Instances Optimization

Aside from getting a nice green console and configuring any exclusions that make sense for your use case, configuring trusted advisor reports is free and can help identify issues. To configure the reports:

  1. Open the Trusted Advisor console.
  2. Click preferences.
  3. Check a recipient checkbox.
  4. Click save preferences.

These reports contain updated Trusted Advisor results and cost-savings estimates (for RIs typically).