Dynamic Alerts for AWS IAM Configuration Changes

This blog is an improved version of this AWS blog post by Will Kruse, In his blog post, he walks us through how to set up CloudWatch alarms on IAM configuration changes. Here’s a quick overview of the setup that was suggested on how AWS usage ends up triggering a CloudWatch alarm. The users of your AWS account make calls to IAM (and other AWS services), and a record of these calls is included in your CloudTrail logs (these records are called “events”). CloudTrail publishes these log events to your CloudWatch logs. CloudWatch allows you to run a filter on these events and generate a CloudWatch metric on any matches. It’s up to you to define when these metrics trigger an alarm, but when enough events occur in a specified time period, you will receive an alert either via an SNS topic or email.

The following images illustrates the process:

The Problem:

The email alert received contains info related to the Cloudwatch metric that reached its threshold and raised an alarm but nothing about the actual log event or AWS activity which caused the alarm. You will have to go through your logs to find out what happened.

The following image shows the info contained in the email received:

Alarm Details:

- Name:                   	IAMAuthnAuthzActivityAlarm

- State Change:           	INSUFFICIENT_DATA -> ALARM

- Reason for State Change:	Threshold Crossed: 1 datapoint (1.0) was greater than or equal to the threshold (1.0).

- Timestamp:              	Monday 26 January, 2015 21:50:52 UTC

- AWS Account:            	123456789012

Threshold:

- The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 1.0 for 300 seconds.

Solution:

We could improve the same setup with minute changes to receive email alerts with more insightful information like the action performed, user identity details, request parameters, IP address, mode of access, etc.

The only change we’re gonna do is instead of using a metric and Cloudwatch alarm we stream the Cloudwatch logs to a Lambda function using a subscription filter. This will allow us to pass the actual log event to the function in the event object. We could parse this information into the structure needed and then pass it to the same SNS topic or send an email alert using SES.

Step-1: Stream Cloudwatch log events to Lambda function

Let us say your Cloudtrail is logging events to /Cloudtrail/logs log group in Cloudwatch, you can stream them to a lambda like this:

Note: You can have a subscription filter pattern to make sure only IAM specific logs events are streamed to the Lambda function and ignore others.

Step-2: Add Subscription Filter Pattern and start streaming

Important: The blank spaces in filter patterns are for clarity. Also, note the use of outer curly brackets and inner parentheses.

Monitor changes to IAM:
If you are interested only in changes to your IAM account, use the following filter pattern:

{ ( ($.eventSource = "iam.amazonaws.com") && (($.eventName = "Add*") || ($.eventName = "Attach*") || ($.eventName = "Change*") || ($.eventName = "Create*") || ($.eventName = "Deactivate*") || ($.eventName = "Delete*") || ($.eventName = "Detach*") || ($.eventName = "Enable*") || ($.eventName = "Put*") || ($.eventName = "Remove*") || ($.eventName = "Set*") || ($.eventName = "Update*") || ($.eventName = "Upload*")) ) }

This filter pattern will only match events from the IAM service that begin with “Add,” “Change,” “Create,” “Deactivate,” “Delete,” “Enable,” “Put,” “Remove,” “Update,” or “Upload.” For more information about why we’re interested in APIs matching these patterns, see the IAM API Reference.

Monitor changes to authentication and authorization configuration:
If you’re interested in changes to your AWS authentication (security credentials) and authorization (policy) configuration, use the following filter pattern:

{ ( ($.eventSource = "iam.amazonaws.com") && (($.eventName = "Put*Policy") || ($.eventName = "Attach*") || ($.eventName = "Detach*") || ($.eventName = "Create*") || ($.eventName = "Update*") || ($.eventName = "Upload*") || ($.eventName = "Delete*") || ($.eventName = "Remove*") || ($.eventName = "Set*")) ) }

This filter pattern matches calls to IAM that modify policy or create, update, upload, and delete IAM elements.

After you finished with filter patterns proceed to the next step, review everything then start streaming to the lambda function.

Step-3: Lambda function code

Add this python code snippet to your lambda function and parse the log event to any structure needed. Maybe even use it to compose a readable message for convenience.

lambda_function.py

import boto3
import json
import gzip
import base64

SNS_CLIENT = boto3.client('sns')
SNS_TOPIC_ARN = 'arn:aws:sns:us-west-2:123456789012:YOURSNSTOPICNAME'

def lambda_handler(event, context):

	data_str = gzip.decompress(base64.b64decode(event['awslogs']['data']))
	data = json.loads(data)
	logs = data['logEvents']
	
	records = []
	
	for log in logs:
		
		# add parsing logic if needed
		record = json.loads(log['message'])
		records.append(record)
	
	payload = json.dumps(records)
	print(payload)
	
	# Push payload to SNS
	SNS_CLIENT.publish(
		TopicArn = SNS_TOPIC_ARN,
		Subject = 'IAM Changes Alert',
		Message = payload
	)

Conclusion

This way we can add a new layer of security to our AWS accounts, even if you don’t have a solid response time to unexpected events at least you know where to start troubleshooting.

I hope it was helpful, thank you!

This story is authored by Koushik. He is a software engineer specializing in AWS Cloud Services.