Simple logging with Amazon CloudWatch Logs

If you don't want to run your own ELK stack and want to use an Amazon style replacement for monitoring logs you can use CloudWatch Logs. You can use Amazon CloudWatch to monitor and troubleshoot using any log files that exist on your instances. These log files can be monitored in near real-time. You can monitor logs for specific phrases, values, or patterns and set a CloudWatch Alarms. For example, you could set an alarm on the number of errors that occur in your system logs. You can also view graphs of web request latencies from your application logs.

Another benefit of CloudWatch Logs is the fact that the log file does not need to be held on the instance for an extended duration. So you can provision instances with lower disk space, or not worry about logs filling up by being more aggressive with log rotation on the instance.

Cost of CloudWatch Logs is fairly reasonable at $0.50 per GB ingested and $0.03 per GB archived per month

So, to get started.

You'll need to ensure that the instance you're launching / running has access to a role policy that permits certain log functions. Here's an example policy:

{
    "PolicyName": "LogRolePolicy",
    "PolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}

On the instance (assuming AWS Linux or similar with the appropriate repository):

wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
sudo python ./awslogs-agent-setup.py --region us-west-2

This will install the awslogs agent on your instance, and prompt you for input. According to the help menu you can just run the installer.

$ sudo python ./awslogs-agent-setup.py --help
Usage: awslogs-agent-setup.py [options]

Options: -h, –help show this help message and exit -o, –only-generate-config Only generate configuration without installing new bits. -n, –non-interactive Non interactive mode. -r REGION, –region=REGION AWS region. -c CONFIG_FILE, –configfile=CONFIG_FILE Local path, S3 path or http(s) based URL of the CloudWatch Logs agent’s configuration file. -u PLUGIN_URL, –plugin-url=PLUGIN_URL URL of CloudWatch Logs plugin. -p PYTHON, –python=PYTHON The Python interpreter to use. The default is the interpreter that virtualenv was installed with (/usr/bin/python)

So if you wanted to do this manually you would run:

sudo python ./awslogs-agent-setup.py -n –region us-west-2 -c /path/to/file

<h4>CloudFormation Logs</h4>
<p>
    One of the handiest things in the world when you're troubleshooting a
    new CloudFormation template.
</p>
<p>
    To use CloudWatch Logs in CloudFormation, we need to add a
    <code>AWS::Logs::LogGroup</code> resource, with the desired retention
    period. We also would need to install and configure awslog service on
    the instances. That's assuming your instance's IAM profile has the
    permissions discussed above.
</p>
<p>
    For the <code>AWS::Logs::LogGroup</code> resource, the modification is
    fairly easy. You would just add the following code snippet to the
    resource section of your template.
</p>
<pre>

“CloudFormationLogs”: { “Type”: “AWS::Logs::LogGroup”, “Properties”: { “RetentionInDays”: 7 } }

In the AWS::AutoScaling::LaunchConfiguration resource used with your instances you would need to add a configuration set for awslogs. In the configuration set you would add:

  • installation of the awslogs package.
  • creating the required configuration files (/etc/awslogs/awslogs.conf, /etc/awslogs/awscli.conf).
  • starting the awslogs service.

So to install the awslog package in the configuration set you can add:

“packages” : {
“yum” : {
“awslogs” : []
}
}

We would need to create the following configuration files in the configuration set:

“files”: {
“/etc/awslogs/awslogs.conf”: {
“content”: { “Fn::Join”: [ “”, [
“[general]\n”,
“state_file= /var/awslogs/state/agent-state\n”,
“[/var/log/cloud-init.log]\n”,
“file = /var/log/cloud-init.log\n”,
“log_group_name = “, { “Ref”: “CloudFormationLogs” }, “\n”,
“log_stream_name = {instance_id}/cloud-init.log\n”,
“datetime_format = \n”
] ] },
“mode”: “000444”,
“owner”: “root”,
“group”: “root”
},
“/etc/awslogs/awscli.conf”: {
“content”: { “Fn::Join”: [ “”, [
“[plugins]\n”,
“cwlogs = cwlogs\n”,
“[default]\n”,
“region = “, { “Ref” : “AWS::Region” }, “\n”
] ] },
“mode”: “000444”,
“owner”: “root”,
“group”: “root”
}
}

You’ll notice for the /etc/awslogs/awslogs.conf file, you can add files like:

[/var/log/httpd/access.log]
file = /var/log/httpd/access.log
log_group_name = { “Ref”: “CloudFormationLogs” }
log_stream_name = {instance_id}/httpd/access.log
datetime_format =

Add any files you’d like to stream to the console.

We also need to run a command to create a state directory:

“commands” : {
“01_create_state_directory” : {
“command” : “mkdir -p /var/awslogs/state”
}
}

So to start the awslog service in the configuration set you can add:

“services” : {
“sysvinit” : {
“awslogs”    : {
“enabled” : “true”,
“ensureRunning” : “true”,
“files” : [ “/etc/awslogs/awslogs.conf” ]
}
}
}

The section below shows all of the required additions to the resource section of your template to start using CloudWatch Logs.

{
“Resources” : {
“CloudFormationLogs”: {
“Type”: “AWS::Logs::LogGroup”,
“Properties”: {
“RetentionInDays”: 7
}
},
“LaunchConfig” : {
“Type” : “AWS::AutoScaling::LaunchConfiguration”,
“Metadata” : {
“AWS::CloudFormation::Init” : {
“configSets” : {
“install_all” : [ “install_cfn”, “install_logs”]
},
“install_cfn” : {
“files” : {
“/etc/cfn/cfn-hup.conf” : {
“content” : { “Fn::Join” : ["”, [
“[main]\n”,
“stack=”, { “Ref” : “AWS::StackId” }, “\n”,
“region=”, { “Ref” : “AWS::Region” }, “\n”
]]},
“mode”    : “000400”,
“owner”   : “root”,
“group”   : “root”
},
“/etc/cfn/hooks.d/cfn-auto-reloader.conf” : {
“content”: { “Fn::Join” : ["”, [
“[cfn-auto-reloader-hook]\n”,
“triggers=post.update\n”,
“path=Resources.WebServerInstance.Metadata.AWS::CloudFormation::Init\n”,
“action=/opt/aws/bin/cfn-init -v “,
"         –stack “, { “Ref” : “AWS::StackName” },
"         –resource WebServerInstance “,
"         –configsets install_all “,
"         –region “, { “Ref” : “AWS::Region” }, “\n”,
“runas=root\n”
]]}
}
},
“services” : {
“sysvinit” : {
“cfn-hup” : {
“enabled” : “true”,
“ensureRunning” : “true”,
“files” : [
“/etc/cfn/cfn-hup.conf”,
“/etc/cfn/hooks.d/cfn-auto-reloader.conf”
]
}
}
}
},
“install_logs”: {
“packages” : {
“yum” : {
“awslogs” : []
}
},
“files”: {
“/etc/awslogs/awslogs.conf”: {
“content”: { “Fn::Join”: [ “”, [
“[general]\n”,
“state_file= /var/awslogs/state/agent-state\n”,
“[/var/log/cloud-init.log]\n”,
“file = /var/log/cloud-init.log\n”,
“log_group_name = “, { “Ref”: “CloudFormationLogs” }, “\n”,
“log_stream_name = {instance_id}/cloud-init.log\n”,
“datetime_format = \n”,
“[/var/log/cloud-init-output.log]\n”,
“file = /var/log/cloud-init-output.log\n”,
“log_group_name = “, { “Ref”: “CloudFormationLogs” }, “\n”,
“log_stream_name = {instance_id}/cloud-init-output.log\n”,
“datetime_format = \n”,
“[/var/log/cfn-init.log]\n”,
“file = /var/log/cfn-init.log\n”,
“log_group_name = “, { “Ref”: “CloudFormationLogs” }, “\n”,
“log_stream_name = {instance_id}/cfn-init.log\n”,
“datetime_format = \n”,
“[/var/log/cfn-hup.log]\n”,
“file = /var/log/cfn-hup.log\n”,
“log_group_name = “, { “Ref”: “CloudFormationLogs” }, “\n”,
“log_stream_name = {instance_id}/cfn-hup.log\n”,
“datetime_format = \n”,
“[/var/log/cfn-wire.log]\n”,
“file = /var/log/cfn-wire.log\n”,
“log_group_name = “, { “Ref”: “CloudFormationLogs” }, “\n”,
“log_stream_name = {instance_id}/cfn-wire.log\n”,
“datetime_format = \n”,
“[/var/log/httpd]\n”,
“file = /var/log/httpd/*\n”,
“log_group_name = “, { “Ref”: “CloudFormationLogs” }, “\n”,
“log_stream_name = {instance_id}/httpd\n”,
“datetime_format = %d/%b/%Y:%H:%M:%S\n”
] ] },
“mode”: “000444”,
“owner”: “root”,
“group”: “root”
},
“/etc/awslogs/awscli.conf”: {
“content”: { “Fn::Join”: [ “”, [
“[plugins]\n”,
“cwlogs = cwlogs\n”,
“[default]\n”,
“region = “, { “Ref” : “AWS::Region” }, “\n”
] ] },
“mode”: “000444”,
“owner”: “root”,
“group”: “root”
}
},
“commands” : {
“01_create_state_directory” : {
“command” : “mkdir -p /var/awslogs/state”
}
},
“services” : {
“sysvinit” : {
“awslogs”    : {
“enabled” : “true”,
“ensureRunning” : “true”,
“files” : [ “/etc/awslogs/awslogs.conf” ]
}
}
}
}
}
},
“Properties” : {
“UserData”       : { “Fn::Base64” : { “Fn::Join” : ["”, [
“#!/bin/bash -xe\n”,
“yum update -y aws-cfn-bootstrap\n”,
“/opt/aws/bin/cfn-init -v “,
"         –stack “, { “Ref” : “AWS::StackName” },
"         –resource LaunchConfig “,
"         –configset ascending”,
"         –region “, { “Ref” : “AWS::Region” }, “\n”,
“/opt/aws/bin/cfn-signal -e $? “,
"         –stack “, { “Ref” : “AWS::StackName” },
"         –resource LaunchConfig “,
"         –region “, { “Ref” : “AWS::Region” }, “\n”
]]}}
}
}
}
}