Biggest re:Invent 2024 serverless announcements
|
So that's it for this year's re:Invent. Werner delivered his usual insightful keynote and shared some key lessons in dealing with complexity and warning signs to look out for. As Werner said, the number of moving parts is not a good measure of complexity on its own. And that's something so many people get wrong about serverless - that they judge complexity by the no. of components on an architecture diagram. Ok, on with the serverless announcements. Aurora DSQL (pronounced "dee-sequel") (Preview)This is, by far, the biggest announcement of re:Invent 2024. Aurora DSQL is a new serverless, distributed, PostgreSQL-compatible database. You can offload the complexity of ensuring your data is in sync across multiple regions. It's built upon globally synchronized clocks that are precise to within a few microseconds. It combines a high level of consistency guarantee with strong performance by allowing each region to operate independently without contention for most operations (except upon commit of a transaction). The engineering behind Aurora DSQL is seriously impressive. You can get a glimpse of its inner workings during Werner's keynote here. Session DAT424 by Marc Brooker, the brains behind Aurora DSQL, is available on YouTube. Watch out for the recording for session DAT427, where Marc gave a deep dive on Aurora DSQL internals. Marc has also shared a 4 part series on Aurora DSQL on his personal blog. Highly recommended! However, Aurora DSQL is still in preview. At the time of writing, many PostgreSQL features are not supported, including, notably, foreign key constraints. There is also a long list of known issues, and the pricing structure has not yet been published. Aurora DSQL is designed for multi-region, active-active workloads, but you can also use it in a single-region mode. Making it a potential drop-in replacement for Aurora Serverless v2. However, now that Aurora Serverless v2 can scale to zero (covered in our pre:Invent post), is there really an argument for Aurora DSQL for single-region workloads? This is not clear to me yet, but I will update you once I have a clearer mental model of where Aurora DSQL fits in. DynamoDB global table multi-region strong consistency (Preview)Aurora DSQL's cross-region replication technology is also available for DynamoDB global tables. It's not a free lunch. Strongly consistent global tables have higher write and strongly-consistent read latencies. It's a trade-off between latency and consistency. If an item is simultaneously updated in two regions, then one of the write operations will fail with a "ReplicatedWriteConflictException" exception. S3 TablesS3 Tables is a new type of S3 bucket with native support for Apache Iceberg. It's specifically optimized for analytics workloads and lets you query tabular data (e.g. Parquet files) up to 3x faster, with 10x higher throughput compared to self-managed tables. If you use Athena, Redshift, EMR, or QuickSight to query tabular data in S3, you should move your data to S3 Tables. Here's how it compares to S3 standard on price: Notice that there is additional data monitoring and compaction costs. These are inline with the likes of S3 intelligent tiering. For the query performance improvements, the pricing makes sense. S3 Metadata (Preview)When enabled, S3 will collect system-defined metadata and user-defined metadata about your objects and make them queriable through the likes of Athena, Redshift, EMR and Apache Spark. It makes easy to find objects by tag, created date, size, content type, storage class and so on. There is a cost of $0.45 per million object updates. OpenSearch can query CloudWatch Logs directlyI haven't tried this myself, but this is a big deal for a lot of companies. You no longer need to duplicate log data to OpenSearch in order to query them. You just need to set up a data source in OpenSearch Discover and can start querying the logs in-place. Pretty sweet! EventBridge and Step Functions can access private APIs through PrivateLink and VPC LatticeTutorial Many AI updatesNew multi-modal Nova foundational models Bedrock supports multi-agent collaboration Guardrails for Amazon Bedrock can now detect hallucinations Guardrails supports multi-modal detection of image content Bedrock intelligent prompt routing Bedrock model evaluation now includes a LLM-as-a-judge |
Master Serverless
Join 15K readers and level up you AWS game with just 5 mins a week.
I recently shared six event versioning strategies for event-driven architectures [1]. In response to this, Marty Pitt reached out and showed me how Orbital [2] and Taxi [3] use semantic tags to eliminate schema coupling in event-driven architectures and simplify the schema management. It's a novel way to manage schema evolution, and I want to share what I learnt with you. Problems with Schema Coupling In an event-driven architecture, event consumers are typically coupled to the schema of the...
Last week, we looked at 6 ways to version event schemas [1] and found the best solution is to avoid breaking changes and minimise the need for versioning. But how exactly do you do that? How can you prevent accidental breaking changes from creeping in? You can detect and stop breaking changes: At runtime, when the events are ingested; During development, when schema changes are made; Or a combination of both! Here are three approaches you should consider. 1. Consumer-Driven Contracts In...
Synchronous API integrations create temporal coupling [1] between two services based on their respective availability. This is a tighter form of coupling and often necessitates techniques such as retries, exponential delay and fallbacks to compensate. Event-driven architectures, on the other hand, encourage loose coupling. But we are still bound by lessor forms of coupling such as schema coupling. And here lies a question that many students and clients have asked me: “How do I version my...