Debugging AWS Lambda functions with Thundra
When I first wrote about Serverless Architectures back in 2016 I described one of the “implementation drawbacks” as being the lack of ability to debug AWS Lambda functions in a cloud environment. In my 2018 update of the article I was still unhappy about this. Now, in 2020, I'm glad to report progress, thanks to cloud tools company Thundra.
Thundra makes a number of different tools for monitoring and introspecting code that you deploy to the cloud. This code can be in various environments - e.g. Functions-as-a-Service (like AWS Lambda) or containers. In this article I'm going to introduce you to their online debugging tool for AWS Lambda.
Thundra actually started their work on Lambda function debugging a couple of years ago, and they approached me then. At the time I wasn't too interested in what they'd built since it was an offline recording and analysis tool, as opposed to a true interactive debugger. This offline tooling has its uses, but it didn't solve my concern. When Thundra told me earlier this year that they now had online debugging, integrated with a standard IDE, I was intrigued. Had they pulled off something even AWS so far haven't done?
The answer is yes! I can debug AWS Lambda functions, running in AWS’ real Lambda environment, live from an IDE on my laptop. Also this requires no code changes!
Let's dig in.
What does it feel like?
To try out Thundra's debugger I decided to use one of the examples from my new book - “Programming AWS Lambda” . This book is about building Lambda-based applications using Java. Thundra supports Java functions, and other languages too. My workbench is the “API Example” from chapter 5 - the example source code is available on Github here. I also decided to use Jetbrains’ IntelliJ - my preferred Java IDE.
Once you've got everything configured you can set a breakpoint and launch Thundra's debugging plugin. Then you call your Lambda function through whatever means you like - here I called it via API Gateway - and lo and behold a few seconds later your breakpoint lights up, and you can interrogate the environment of your Lambda function. Magic!
Here I've set a breakpoint on the first line of code in my function. We can look at the request object passed in to our handler, in this case by API Gateway, and its value. Here I've set the
limit query string parameter to be the value
10, and I can see this same value embedded in the API Gateway
I can then resume the code and see what happens next.
Now we can look at more of the state of our function - in this example, after calling DynamoDB. Again - this isn't a pre recorded execution, this is happening live. There's a delay here (understandably - my code is running a couple of thousand miles away from where I'm sitting) but this is impressive stuff.
Setup and configuration
To get this working there are 3 main tasks: get an “auth key”, update your deployment configuration, and install and setup your IDE plugin.
To get your auth key, first sign up for an account at thundra.io if you don't already have one, and once signed in go to the debugger settings page to get an auth key. This auth key will be the link between your local debugger and your remote code.
Next - your deployment configuration. In our book we use AWS SAM as our deployment tool, and Thundra have documentation for this case. The easiest way for me to describe what you need to do is to show a diff vs the original template. (This is also available as a gist here).
There are a few things here you need to pay careful attention to. First, I use Thundra's custom Lambda layer (
Layers). Here I'm using version 42 of it, but by the time you read this that may have changed. Next I tell SAM that I want to use the custom runtime in the layer (
Runtime: provided). I discuss that more in the caveats section, in a moment. I set a new element for
Environment Variables - the auth token I copied from the Thundra website. And finally I increase my
Timeout, since we want to keep the function alive while we're debugging it - I set it for 5 minutes here for the sake of the demo, but you should feel free to set it to longer if you'd like. With all these updates in place deploy your function - usually to a development or testing environment, rather than production.
Then set up your IDE - this is actually pretty simple with IntelliJ. First, install the plugin from the marketplace. Then create a new “Run / Debug Configuration”, with the Thundra Debugger plugin. Add your auth token in its place in the configuration's dialog, and then hit “OK”.
With everything in place, start a Debug session in your IDE. Once it's running, execute your function, either via it's configured trigger (e.g. for an API Gateway-triggered Lambda then make a request to the API), or directly using the web console or CLI. With a little luck your IDE will connect to your Lambda function in the cloud (via some coordination with Thundra's servers). The first time you do this at least it might take 10 - 20 seconds or so.
Caveats / Limitations
I’m normally not one to get excited about a tool like this since I find they typically make too many compromises to give generally useful information. However, there are surprisingly few limitations to Thundra’s approach. Often third party integrations require some amount of custom code, or at least custom libraries to be included in the function's build-time distributable, getting in the way of all of my other code. Thundra has pulled off this debugger without requiring that.
There are a few things worth keeping in mind though. First you're using a third party vendor here - not AWS - and you're going to have to run a custom Thundra runtime in your AWS environment. For some security conscious companies this can be a hurdle. The good news is that this is typically only something you're going to run in a development or testing environment.
Second is that the debugger is using a custom runtime, and so when you use the debugger you won't be using precisely the same environment as production code. However, it's very nearly the same. Thundra's custom runtime actually uses the Java 8 VM that comes standard with all Lambda environments, just with some configuration updates.
And that last point means one other thing to consider - this only supports Java 8 at time of writing, and so if you're writing a Java Lambda it will need to be able to run in Java 8. Of course if you're using a different language then check out Thundra's documentation to see whether it's supported.
What to use this for
Typically when I'm trying to debug problems with a Lambda function I write a unit test or functional test that runs purely within my IDE. If you want to know more of how John and I do that, then see chapter 6 of our book! However with such tests I'm making assumptions about the Lambda function's runtime environment, and how the function integrates with other systems and services. Based on my experience of writing Lambda and AWS code these assumptions are often correct, but not always.
When those assumptions aren't correct it's expressed by me seeing different behavior than I'd expect, from a view outside my Lambda function. For these situations an online debugger is very useful. It allows me to look inside my Lambda function, and see how it is integrating with its environment and downstream systems. Sure, I can use logging for some of this, but for particularly tricky scenarios it might be hard to know what to log. Also, I might not be able to accurately simulate the responses of those other systems when running locally, perhaps because of a nuance of my Lambda function's role, or some other part of its environment.
Congratulations to Thundra - they've pulled off an impressive feat. Next time I'm scratching my head wondering what on earth is going on inside my Lambda function I'll know where to turn.