Examining the AWS Serverless Application Repository
AWS announced a new service — the Serverless Application Repository (SAR) — at re:Invent 2017. The general goal of SAR is to make it easier to distribute, and consume, applications that have been developed using AWS Serverless products, like Lambda. It was launched to the public in February and I've been digging into it since. This article describes what SAR is (and isn't), examples of SAR in the wild, and some limitations and ‘gotchas’ of SAR. I close out with some opinions of its value and future.
What is the Serverless Application Repository?
For those with a decent amount of experience of AWS, the quickest way to describe SAR is that it is a searchable catalogue of cross-region-deployable CloudFormation templates.
For the rest of you, I'll explain what this means.
A couple of great things about Serverless technologies is that applications using Serverless are incredibly quick to develop, and in low-usage scenarios cost very little or nothing to run. This makes Serverless a good way to build many small components that can be used to glue various other parts of an application, or production environment, together.
One question that follows this is how best to deploy Serverless apps. There are a plethora of deployment tools now available, from the original community-produced Serverless Framework, through to Amazon's own Serverless Application Model (SAM). SAM itself is built on top of AWS CloudFormation — Amazon's ‘Infrastructure-as-Code’ language and deployment service. What SAM does is allow the deployment needs of a Serverless app to be defined relatively concisely (think defining some Lambda functions, an HTTP interface, some security rules, maybe a small Dynamo database in a page or so of YAML), in a way that can be easily expanded to use the rest of Amazon ecosystem.
A second question then comes up — once a Serverless app is defined, how does one distribute it? Often times for both public and private Serverless apps and components, a source control repo is used. If, as an application provider, you don't want to force the users of your app to rebuild your application, you may provide a pre-made CloudFormation package, but this requires a different package per AWS region. Not ideal. And even then CloudFormation isn't, by itself, a great interface to consume third-party components.
Enter the Serverless Application Repository. SAR provides a way for application publishers to distribute an application defined using SAM; and for application consumers to find such applications, and more easily deploy such applications to their own AWS accounts. Publishers can distribute applications to one AWS Region (as of writing, specifically us-east-1 or -2), and consumers can deploy to any (supported) region they like.
To make this clearer, let's look at some examples.
First of all here's one I built myself — CodePipeline Slack Notifier . This is a small app that listens for CloudWatch events generated by CodePipeline (Amazon's deployment pipeline tool), and when it hears one will send a message to a Slack Channel.
I first published this app last year, via its Github repository. This worked fine, but required users of the application to build the app, and then use CloudFormation to deploy it, which can be a little complicated for those that aren't used to either the underlying development tools, or to CloudFormation.
Last week I launched a new version of this app, and made it installable via the Serverless Application Repository here. Now you can easily deploy the app without having to download the Github repo, or use CloudFormation (at least not to start with.)
A similar example to this is Stelligent's Pipeline Dashboard, which also provides visualization around CodePipeline.
I'm not going to describe too much how to use SAR in this article, but here are some links to set you on your way:
What the Serverless Application Repository isn't
As I started getting my head around SAR it was interesting to realize what it isn't.
First of all — it is absolutely not a catalogue, or marketplace, of services that you can integrate remotely from your AWS account. As an application consumer, SAR applications will run inside your own AWS account, and therefore while you may not have developed a SAR app, you will need to operate it. I'll get onto this in more depth in a moment, but it's crucial to understand that SAR is a very different beast to, say, the Heroku marketplace.
Also, while SAR is a way to find and install apps, once they are running you'll need to use the underlying technologies (CloudFormation + whatever services the SAR app uses) in order to operate them as a consumer.
Finally, SAR isn't just the “Lambda Blueprints” catalogue rebranded — it's much more than that. SAR apps can consist of multiple (or no) Lambda functions, can create other resources, and (more importantly) SAR offers self-service for publishers to distribute their applications
Limitations / Warnings / Gotchas
SAR is a brand new service, and in typical AWS ‘first release’ fashion, is both (rightly) limited in scope and has some rough edges. Caveat emptor, etc. But here are a few things that you may want to be aware of.
As a SAR Consumer
First, let's talk Operations. As I mentioned above, a SAR application runs in your own AWS account, so you have to be aware of various operational needs and concerns of the application. From a security standpoint, the SAR UI will tell you what IAM permissions the application needs, which is a good start. But what isn't shown are things like service usage quantities, log configuration, metrics you may want to track, etc. That service usage concern is especially important for elements that have cross-account limits, e.g. Lambda concurrency. As you start using SAR apps in production at scale, you should request operational documentation from the application developer. Statically you may be able to discern a lot from the application template file (also shown in the SAR UI) but this won't tell you about other runtime characteristics.
Next, let's consider security. There isn't too much validation performed by AWS, as far as I can tell, on SAR apps. For a publisher to put an application in SAR they need to have an AWS account, and the FAQ says “All applications published by AWS are vetted for license adherence and code quality. Applications published by third parties are validated for correct use of permissions…" but that doesn't appear to be an extensive check. This means that a reputable organization distributing SAR apps could make a mistake, and affect your AWS account in undesirable ways. Disreputable organizations and individuals could do things that are far worse.
Digging into this a little more, SAR apps need to define which resources they need access to within their SAM file, and they won't be given access to resources other than these. They also need to be explicit about any IAM permissions / policies they need. In many cases this should provide enough protection, assuming that consumers pay close attention to what a SAR app is requesting.
However, having access to a resource, and only doing with that resource what you say you're going to do, are two different things. For consumers that want to dig in further to what an app does they can in theory examine the source for an application version, as linked in the documentation of a SAR app, however there's no guarantee that that source code represents the actual logic in the deployed package — AWS are trusting publishers to keep source and package in sync.
In other words when using SAR apps you may want to consider the following:
- Look carefully at the permissions and resources a SAR app is requesting. If it requires access to an S3 bucket, make sure it only has the level of access you'd expect. If you store anything else in that S3 bucket other than the data needed by the SAR app, consider separating those locations.
- Remember that looking at resources doesn't tell the whole story — e.g. Lambda functions can make HTTP calls to external services anywhere on the public internet without any extra granted permissions.
- Think about only trusting SAR apps from organizations you yourself trust. For others, consider using the source code link and rebuilding yourself when using in production.
- When using the SAR search console for production apps in a data sensitive context, you may, instead, want to only use direct links to SAR applications from an external website — there is nothing in the SAR UI right now to prove that a SAR app comes from who you think it does.
- Consider more lenient use of SAR only in development accounts.
Wrapping up what we've talked about so far in this section — there are significant operational and security requirements on the consumer to understand what a SAR app is doing — AWS give you some tools to help here, but you should not consider SAR apps an operational black box.
OK, let's move on, and consider UX.
There are 2 different UIs for searching for, and picking, SAR apps — through the SAR home page at https://serverlessrepo.aws.amazon.com/applications , and through the Lambda section of the AWS console (‘Create Function’, ‘Serverless Application Repository’). Somewhat strangely the first of these is not part of the regular AWS console. This is useful, in a way, since it means you can browse the SAR catalogue without logging into to the console … but once you want to deploy something you'll need to login anyway.
Once you've picked a SAR app to deploy from either of these locations, you'll be (re)directed to the Lambda console. Using the Lambda console to deploy a SAR app is, frankly, confusing, since what you're deploying is actually a CloudFormation application which will probably (but not necessarily) contain one or more Lambda functions. There is a SAR section of the AWS Console but that's currently only used for publishers.
Once you've kicked off installation of your SAR app there's no other SAR specific UI. You'll see which apps and resources were installed via SAR by their name in CloudFormation, Lambda, etc., but that's pretty much it.
You are able to upgrade SAR apps to new versions, and the way to do this is by simply re-running the app installation (and providing parameters again), by using the same Application Name you've already used previously for an installed SAR app. Remember — an application's resource and permission requirements may change across versions (although you won't be notified of such), so make sure to re-validate them on every upgrade.
Overall, while the SAR Consumer UX is functional right now, it's not obvious how to use it, and has varying levels of abstraction depending where in the lifecycle you are. It needs a lot more work, but AWS being AWS, I'm sure it will get it.
Finally, while there is support for consuming apps via the CLI, it's currently a little rudimentary.
As a SAR Publisher
One of the first things you may notice when publishing a SAR app is that cross-region syncing is slower than you might expect. As I mentioned earlier, one of the benefits to SAR is that a consumer can install a SAR app to any region they choose, as long as you as a publisher create the SAR app in us-east-1 or us-east-2 . However the cross-region syncing from us-east-* out to the other regions is noticeable. Last week it took between 10 and 30 minutes for a new app, or new version of an app, to appear for a consumer in us-west-2 after I'd created it as a publisher in us-east-1. For newly created applications trying to access sooner than this may show as an error, for new versions there'll be a delay in the version being available in other regions.
At present public SAR applications must be open source, with a valid ‘SPDX’ license, and providing source code. As such you can't use SAR to distribute proprietary code publicly, and nor does SAR have any billing mechanism. This restriction (despite some currently confusing UX and documentation) doesn't exist for private applications, i.e. those that don't appear in the search catalog and are meant only for organization-internal use.
SAR is limited to using SAM as the deployment definition mechanism, and more-over only a specific set of AWS resources can be used, as described here. AWS have said they are open to expanding this set, but for now you should keep this in mind. For example when producing the SAR version of the CodePipeline Slack Notifier, the original version of my SAM template was not SAR-valid, and I needed to update the template to use it with SAR.
One thing that's more of a ‘not gotcha’ (that I misunderstood at first) — while you need to provide the application package to the SAR service via S3 (as is performed by the
aws cloudformation package command before you call SAR), that S3 bucket is not what is used by consumers. Instead, at SAR publication time, AWS copies the package from your bucket to a distribution repository. Among other things this means that you aren't charged for any data transfer charges when consumers download your app. It does mean that your package bucket needs to be accessible to SAR, and how to do that is described here.
In theory SAR can optionally be used to distribute cross-account, within an organization (i.e. not publically) . I haven't tried this out, but at this time such functionality is only supported through the CLI via the aws serverlessrepo put-application-policy command, and not via the web UI.
Finally, and this is a small thing, the public URL for a SAR app is … umm … a little verbose. My CodePipeline Slack Notifier's URL is https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:392967531616:applications~CodePipelineSlackNotifier . You may want to use a URL shortener when providing direct links! Even better would be for AWS to make _‘https://serverlessrepo.aws.amazon.com/app/CodePipelineSlackNotifier'_ a valid url.
Thoughts about the Future of SAR
I think SAR, despite the limitations I've described above, is a positive step forward. Serverless is a huge mind-shift in how we develop, operate, distribute and even conceptualize applications and application components. As an industry we're still getting our heads around what Serverless Application Architecture looks like, and also what good ways are to deploy and distribute apps.
SAR is, if nothing else, an interesting experiment in advancing knowledge in all of these areas, and with some refinement could become much more than that — an excellent tool for rapid development of systems.
- 3/15/18 — Updated section on open source restriction, mentioning that private SAR apps don't need to be open source. Thanks Tim Wagner .
- 5/31/18 — Removed section about CLI not working for publishing — I've successfully used it now.#### Upcoming Workshops
I hope you enjoyed this article! You'll find plenty more of where this is coming from in workshops that John and I are giving this summer, why not pay us a visit? We're teaching the following courses:
Serverless Continuous Deployment, London, UK on May 15 . Always fun to get back to my homeland!
John will be giving another Serverless Masterclass in Rome, Italy, on June 5 . I'm not jealous of this trip. At all. Ahem
And finally I'll be teaching Mastering Continuously Deployed Serverless Applications, in San Jose, CA on June 11 . This last workshop is part of O'Reilly's Velocity conference, and I'm thrilled to be hosting, and speaking in, a great Serverless track that will also be happening that week. I hope to see some of you there.
If you're interested in any of these, or would like to chat to us about our consulting services, or please contact us.
And if you're just looking for some more interesting reading, why not check out our free e-book with O'Reilly — What is Serverless?