Revisiting AWS Lambda Runtime Performance
The more things change, the more they stay the same.
Back in February, I wrote an article titled “The Occasional Chaos of AWS Lambda Runtime Performance”. The performance benchmark results in that article showed that while worst-case performance was indeed proportional to a Lambda's memory configure, actual performance could be much better than that worst case. In fact, over the two day duration of the benchmark, the 128MB Lambda was often just as performant as the 1.5GB Lambda.
I drew two conclusions from those observations. First, if consistent performance is important for a particular Lambda function, the best way to achieve that is to crank the memory setting all the way up to 1.5GB. Secondly, benchmarks from Lambda functions must be gathered over many hours or days, rather than just a few minutes or a few hundred invocations.
Same benchmarks, new results
With those conclusions in mind, I decided to re-run that same Lambda benchmark again, four months later, to see if anything has changed. As I found, the more things change, the more things stay the same.
As before, all of the code used for these benchmarks can be found on Symphonia's Github page. This time around, I've also added a nice Serverless Application Model (SAM) template, to ease deployment. Other than that, I've made no substantive updates to the code.
The benchmark in my previous article was run over the course of two days in February, 2017. For this article, I ran the same benchmarking code over the course of two weeks, across two different AWS regions (us-east-1 and us-west-2). All told, each Lambda function executed over 5000 times.
Just north of California, dreamin’
As you can see from the chart, during this period of time, Lambda performance in the Oregon region (us-west-2) was much more stable than in February. In fact, there are no instances in which a lower-memory Lambda is faster than a higher-memory Lambda (during the same interval). However, performance of individual Lambdas did vary, by as much as 20% even when cold starts are ignored.
The chaotic performance of February has been followed by two weeks of smooth sailing, so we can go back to assuming that Lambda performance is always consistent. Right?
Wrong. Let's check out a different region.
Heading east, into the fray
This second chart is a little more interesting (although still not nearly as exciting as February). The chart shows Lambda performance in the Northern Virginia region (us-east-1), over the same two week time period as the previous chart. The first obvious incident in the graph occurred on June 19th (UTC) — it's unclear whether this was a more pervasive issue or something isolated to a few specific instances of the benchmarking Lambdas. In any case, performance of the 128, 256, 512, 1024 and 1280 MB Lambdas was dramatically affected for several hours, while the 768 and 1536 MB Lambdas continued executing normally.
The second incident on this chart is the publicly acknowledged Lambda outage that occurred early on June 23rd (UTC), in which Lambdas in the Northern Virginia region were affected. As you can see, the benchmarking Lambdas didn't stop executing, but their performance jittered a bit.
In the coming months, I'll be regularly revisiting Lambda runtime performance. These benchmarks will continue to run continuously in the Northern Virginia and Oregon AWS regions, and we've mostly automated the metrics processing and charting pipeline to make this information more readily accessible.
The same conclusion
As before, my conclusions remain largely unchanged. Consistent Lambda performance is more likely with a higher memory setting. Most importantly, benchmarks for individual Lambda functions should be run over several hours or days, rather than just a few minutes or a few hundred invocations.
Watch this space for more Lambda performance updates in the near future!
John Chapin is a co-founder of Symphonia, your expert Serverless and Cloud architecture consultancy. Symphonia helps companies of all sizes and industries deliver business value at scale through Serverless technology. Learn more at www.symphonia.io.