Books, Elevators, and Serverless Data Platforms

Symphonia
Mike Roberts
Sep 23, 2020 · 6 min read

This is the latest edition of our newsletter. To receive the email version please subscribe here.

Hello from Symphonia,

It's been a tough, and strange year. John and I (Mike) are very grateful to be safe, healthy, and still in business. Living in New York City, we know especially well that not everyone is as fortunate. So first and foremost, we wish you and your families the best both for today, and for the future as we all see through this global crisis.

It's been a long time since we sent out one of these newsletters, and for many of you it will be your first from us - welcome! A lot has happened since our last broadcast, so I'll pick out some highlights. Today I'm going to chat about our new book; elevators; and why data warehouses are cool again.

If you're feeling bookish

Our book, with an O'Reilly animal!

Last time we emailed I mentioned that we'd started writing a book, and I'm very happy to say that it was published earlier this year. "Programming AWS Lambda: Build and Deploy Serverless Applications with Java" is O'Reilly's first full book dedicated to a serverless topic, and as far as we know the only Lambda book written with Java developers specifically in mind. Covering first steps, engineering practices like testing and monitoring, while also discussing more advanced architecture topics, Programming AWS Lambda distills a lot of what John and I have learned over the years.

While Java developers will enjoy the specific examples geared towards that ecosystem, the book isn't just for Java programmers, and you don't need to take our word for it! Jeremy Daly, “AWS Hero”, has this to say:

“This book covers many of the fundamental serverless concepts that developers really need to know. If you happen to be developing in Java, then that’s just a bonus.” (see more of his review here)

We are very happy with the book, and we were also thrilled that “father of Lambda” Dr. Tim Wagner wrote the foreword - thanks Tim!

It's available from all good bookstores, as well as O'Reilly's online learning site. If you'd like more details then take a read of this summary on our blog.

What else have we been up to?

While the book was a big project, the vast majority of our work is still with clients - large and small, across an array of industries. Some of our relationships are now years long, while others are quick projects to help people through a one-off concern.

Typically we start off with a client with an architecture review - either for an existing system that needs some re -steer, or a proposed product that is expanding into technologies new to an organization.

With our longer-term clients our work then branches out in a number of different directions. Three of the types of work we've performed recently are proof of concept delivery; longer-term trusted advisory support for architecture and cloud strategy; and player-coaching embedded on teams.

My old friend and colleague Gregor Hohpe (now an enterprise strategist at AWS) has coined the term over the last few years of “the architect elevator". This is the idea that architects in modern digital enterprises need to be equally able to communicate effectively both with company executives about strategy, and engineers about low-level implementation. This is a pretty good summary of what John and I do.

On the technology side our work has come to be almost entirely focused on AWS. We have a “virtuous cycle” where the more we learn about different AWS services for one client, the more effective we are at helping the next client with the AWS platform. That being said we remain vigorously independent, and are very happy to steer clients away from a particular AWS service if we think it's a bad fit for them.

What have we been excited about recently?

One aspect of our work that's been fascinating to see evolve this year has been serverless data platforms. We've helped three separate clients build capability in this area, just over the last six months. Comprising AWS technologies like S3, Glue, Athena, Step Functions, and more, serverless data platforms take the idea of a serverless data lake based on just S3, and grow it to include data ingestion, ETL, machine learning, and data analysis, all with the economic and time-to-market advantages of a serverless approach.

One of the reasons we've enjoyed working in this area so much is that it is one AWS is adding capabilities to at a rapid pace. One example is the release of “Glue ETL” version 2. Apart from offering significant cost savings and performance improvements, it also makes Glue ETL a far more attractive technology for a much wider range of problems: Serverless data platforms now play nicely with “medium data” and not just “big data”. See John's write-up of Glue ETL V2 on our blog, including descriptions of 40% runtime improvements and 70% cost reductions for one of our clients.

Another example of AWS’ advances in this area is with their Redshift data warehouse database. While this may not sound “serverless”, Redshift has had the “Redshift Spectrum” feature for a while. This allows a cluster to query data located in S3, without the need to load it into the cluster's own storage. The interesting part here is that AWS has been expanding the Spectrum features this summer, including materialized views, and even support for writing data to S3 directly from Redshift. Additionally, Redshift got Data API support a couple of weeks ago, making it easier to integrate Redshift with Lambda.

This is a great new example of an idea John and I have been promoting for years - serverless isn't a binary, all-or-nothing approach. Serverless can instead be a hybrid incremental strategy that organizations lean into, or away from, as their needs change. At the end of the day, serverless done right is about pragmatism. What is the right architecture that solves the organization's business needs, given everything we know today, confident that our solution can happily evolve over time?

That's why we think Gregor's metaphor of the architect elevator is so useful. As architects we can work with executives today to come up with our path forward, work with engineers tomorrow to make that vision a reality, and then re-assess both strategy and implementation in a month's time, knowing that our platform provides us options to switch direction as necessary.

Closing Out

As I said at the top, it's been a tough year. John and I have been modifying how we work with clients, as well as figuring out how to remain happy and effective, all in light of living through a pandemic. Some things are very different - working from home full-time, plus my experiments into producing YouTube training courses.

But fortunately some things don't change - we still get to help clients move their businesses forward, using the best new technologies available. We're always happy to chat to folks about how we might help them, so if you're interested in anything I've talked about in this newsletter then drop us a line at mikeandjohn@symphonia.io.

Until next time,

Mike & John

To receive the email version of this newsletter please subscribe here.