During this week's live Q&A session, a student from the Production-Ready Serverless boot camp asked a really good question (to paraphrase):

"When end-to-end testing an Event-Driven Architecture, how do you limit the scope of the tests so you don't trigger downstream event consumers?"

This is a common challenge in event-driven architectures, especially when you have a shared event bus.

The Problem

As you exercise your system through these tests, the system can generate events that are consumed by downstream systems. These can create a lot of noise for the downstream systems, especially if we use test events that they can't process.

For example, maybe our test events do not contain all the fields, only the ones that we need to exercise our code. Or the event might reference external entities that do not exist (but our system doesn't need to verify).

These often trigger errors and alerts for the downstream systems and make us a bad neighbour!

I have long championed the use of ephemeral environments to allow developers to work on different features in isolated environments.

It's an excellent fit for working with serverless technologies and their usage-based pricing. There's negligible cost overhead for having many ephemeral environments when you are not paying for uptime.

However, ephemeral environments do not directly address the problem at hand. Events generated by end-to-end tests against the ephemeral environments will still cause the undesired side effects downstream.

The Solution

One way to address this problem is to conditionally create a copy of the shared resource (e.g. an event bus) as part of the service stack.

When you create an ephemeral environment, you will make a copy of the event bus (local to the system under test) and use it instead of the shared event bus.

Thus, you can achieve the desired separation between environments and avoid waking up your downstream neighbours!

Related resources, such as IAM roles, resource policies, etc., must also be created conditionally.
​
I've used this approach a lot, and it's relatively easy to implement.

Importantly, it allows teams to develop, deploy, and test their services independently and reduces cross-team dependency, a key indicator of high performance (as noted in Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Gene Kim)

The Implementation

You can implement this solution with any Infrastructure-as-Code tool.

With CloudFormation or tools that are built upon CloudFormation (e.g. SAM, Serverless Framework), you can use CloudFormation Conditions.

I have also created a plugin for the Serverless Framework to make it easier to express conditions like this:

​
With CDK, it's a simple if-else statement.

With Terraform, you can use the count meta-argument, like this:

Other Approaches

Another approach is for everyone to agree that:

• Events generated by tests should include a "is_test" attribute.
• Event consumers should filter out events where "is_test" is true.

I do not recommend this approach because it requires coordination from all participants (both event publishers and subscribers).

A standard abstraction layer is key for this approach to work.

However, implementing consistent behaviour across the board can be challenging, especially if you need to support multiple programming languages and IaC tools.

It only takes one non-conforming participant to break the whole chain.

This approach adds complexity to both event publishers and consumers. Whereas the aforementioned approach only affects event publishers, and event consumers are none the wiser.

However, if most consumers in your system are also publishers, then there may not be much difference in implementation overhead.