Serverless 'Glue' apps in AWS, and sending Slack notifications for Code Pipeline Events
When John and I give our introductory presentations about Serverless we list 3 typical use-cases:
- Web apps / web APIs
- Data Pipelines
- ‘Glue’ apps
This last category is one of the most popular usages of Serverless that we see at present. The low, and precise, infrastructure cost of Serverless makes creating small, one-off, highly specific services a very attractive option. Examples include:
- Providing a proxy / facade over an existing service, with customizations for a specific external partner
- Forwarding log events to an external log visualizer
- Alerting security engineers when certain AWS Account actions are performed
- Custom CloudFormation resources
I want to make it clear that when I say ‘glue’ I'm not referring to the AWS Glue service. That product is relatively recent, and I was using ‘glue’ before the service came along. I haven't come up with a suitably satisfying alternative description at this point!
Building Serverless Glue Apps
When we build Serverless glue apps we typically perform at least the following activities:
- Identify a source system
- Choose a target system to update
- Understand the transformation that needs to happen between (1) and (2)
When we're building Serverless glue apps that run in Amazon Web Services (AWS) the source system is most often an AWS service, perhaps as a proxy for an external system. Lambda event sources are a good source of inspiration here.
The target system may be another AWS service, it may be a non AWS service within our AWS account (perhaps one of our own applications), or a remote service.
The transformation step requires taking the event from the source system, querying its value, and updating the target system accordingly. Typically we use AWS Lambda to perform this transformation but occasionally Lambda is not necessary, and we'll see an example of that later in this article.
Once we understand all three of these activities we're ready to implement our solution, and often that will take less than a day to complete.
In the rest of this article I give an example of creating a Serverless glue app, specifically one that will listen for events from the AWS CodePipeline Service, and then publish a notification to a Slack channel.
AWS CodePipeline Events
AWS CodePipeline is Amazon's Build Pipeline / Continuous Delivery service. At Symphonia we're fans of CodePipeline because it is itself a Serverless service — it doesn't require us to manage any long running build servers — and because it integrates closely with CloudFormation, our current AWS deployment tool of choice.
One useful feature of CodePipeline is that it publishes CloudWatch events whenever various things happen within its flow. Examples are a Pipeline run starting, succeeding, and failing, but you can get a lot more precise than that if you need.
Instances of CloudWatch events aren't viewable on the AWS web console — there's far too many of them for that to be useful. Instead the typical way to use CloudWatch events is to create a CloudWatch Event Rule that is configured for a specific class of event, and this rule will then trigger one or several Targets to process that event.
A very common target type to use with CloudWatch Events is AWS Lambda since that gives us complete programmatic control of what to do with an event. What we do from there is only limited by our imagination!
A useful application of CodePipeline Events in particular is to notify human beings (that's us) that something has happened within our build pipelines. And in this modern age a popular way to do that is through Slack.
I've created a glue app that will capture CodePipeline events and publish notifications to a Slack Webhook. You can view and download the code for that at https://github.com/symphoniacloud/code-pipeline-slack-notifier . I'm not going to explain in this article how to use the app — that's what the README‘s for! But I do want to explain the thinking behind the implementation.
Building the CodePipeline Slack Notifier
The first activity in building this small app is identifying the source events. In this case pretty much everything we need to know is in this documentation page, that I just mentioned above. With the content from that page in mind we can create a simple Lambda function to process those events.
My typical flow at this point in the process is to build and deploy the Lambda using some simple command line tools, and then use the Web Console to configure the event source the first time I use such a source.
The first Lambda code I created in this case was simply to log the inbound event. Sometimes it's not at all obvious from the AWS documentation what the Lambda function is going to receive so logging a real event is useful info to have. An example of such Lambda code is here, and you can do something very similar in your language of choice.
Once I've captured an example event in the logs I can then copy it into a test event within the Lambda Web Console, which is great for debugging.
Next I need to decide what to do with the event. In this case we want to post a message to Slack. I've done this before a few times, so I knew I wanted to use a Slack Incoming Webhook for this functionality. Implementing that is nothing to do with the AWS environment, per-se, so I can figure all of that out in my development environment. Then it's a matter of hooking the code into a Lambda handler, deploying the updated function, and executing it using the test event I created a little earlier.
In theory I should now have a complete working system, so in this case I can trigger a build in Code Pipeline and see what happens. I might have got something wrong, so using CloudWatch Logs for my Lambda function is going to come in useful.
By this point I have a working prototype, and I want to do some amount of hardening. Remember — this isn't a multi-million line application (it's actually significantly less than 50 lines) — so we don't want to go overboard with making this pristine, but it is worth doing things like moving the Slack configuration out to an environment variable and writing a reproducible deployment script.
For AWS we at Symphonia tend to use CloudFormation / SAM for deployment where we can, and this app is an example of that. You can see the SAM file for the Slack notifier here. Honestly, CloudFormation and SAM are something of a hassle from a developer UX point-of-view, but once you've figured out what you need they tend to be the most solid option. And what you learn from building one app you can take forward to the next.
With automated deployment finalized I can tear down the manually deployed version and rely instead on the automated one. Putting the code in source control and adding some documentation finishes the job. If I'm super-keen I can put the app itself in a Continuous Deployment pipeline — whether or not that's necessary depends likely on how often the code is going to change, and how critical it is.
Use the Platform
When getting started with these glue apps it's tempting to do as much work as possible in a Lambda function — this way you're in a familiar environment, can use your tried and trusted unit testing techniques, etc. But often times this isn't the most efficient way to use AWS as a complete Platform. Here are a couple of examples.
First of all, in the Code Pipeline Slack notifier, I don't filter on types of pipeline event, or on what pipelines to notify about. We could perform such filtering very easily in code, but alternatively we can use the power of the Platform to filter events before they even reach our Lambda function. I give some examples in the README of the project.
Another example is where Lambda may not be necessary at all. Paul Duvall from Stelligent has also written an article, and glue app, to process CodePipeline events but his notification target is sending an email. For this he can use Amazon SNS, but targeting SNS can be configured directly from the CloudWatch Event rule, as he shows in the article. I still call this a ‘glue app’ even though there's no actual program code, since it's still a component that performs logic and needs to be deployed and managed.
Why would you lean on the Platform in this way, rather than sticking to regular programming? There are two main reasons. First — cost. Lambda functions are charged by how much time they actually run, so if you can avoid calling Lambda at all, then that's some runtime cost removed. Second — complexity. If you can avoid deploying a Lambda function then that's an extra component that doesn't need to be managed.
Of course there's a trade-off here. You may find that by not using a Lambda function to it's fullest, or at all, then you end up with some gnarly, hard-to-maintain, AWS configuration. In this case it's sometimes worth just using Lambda after all in the name of testing and developer sanity. Where you draw the line is an art, not a science.
‘Glue’ applications are a hugely powerful way of using Serverless techniques in AWS, and other platforms. I showed in this article an example : sending custom notifications to Slack from AWS CodePipeline using a simple, small, Lambda function, and a small amount of SAM deployment configuration.
Why not try out building a new Serverless glue app for your own organization?