Banana For Scale

IaC Testing Methods & Approaches

October 19, 2021

In this article, I would like to evaluate what types of IaC tests can be useful for maintaining our micro-service an a healthy state. Some examples of tests can be categorized in the following groups:

  • Unit Tests
  • Deployment Testing
  • Integration Tests
  • Runtime Tests

Additional considerations are as follows:

  • CI integration
  • Ephemeral (adhoc) environments
  • PR integration
  • Azure DevOps
  • AWS Code Services
  • CircleCI
  • GitHub Actions
  • GitLab CI
  • Google Cloud Build
  • Travis

Let’s get started.

Unit Tests

When we want to test individual functions or pieces of logic in our micro-service, we can use unit tests to accomplish this. Unit tests can be leveraged to validate that the state of our micro-service’s infrastructure is meeting certain conditions. These conditions stem from the various safe and secure configurations needed by the micro-service exist, such as:

  • Naming conventions
  • TLS/SSL configurations
  • Encryption settings
  • Other certain attributes

Additionally catching common mistakes, like forgetting to encrypt storage buckets, or exposing virtual machines unprotected to 0.0.0.0 would be crucial in validating that the infrastructure was actually provisioned in the correct manner, or by preforming runtime tests for application logic that runs inside the infrastructure’s micro-service.

Unit tests evaluate the behavior of your infrastructure in isolation. External dependencies, such as databases, are replaced by mocks to check your resource configuration and responses. It’s possible to use mocks because responses from cloud providers are well known and tested. You already know how, given some parameters, the provider will respond.

Unit tests run in memory without any out-of-process calls, which makes them very fast. Use them for fast feedback loops during development. Unit tests really help you solve problems early in the life cycle of your infrastructure.

A few examples of what you can verify are:

  • Resources are correctly tagged.
  • Instances don’t have an SSH connection open to the Internet.
  • Web site URLs are valid.

When you’re planning your tests, think about using a tool that lets you write your tests in a general purpose language such as Python, Go, TypeScript or C#, rather than in a special-purpose DSL. Standard languages all have well-understood tools and frameworks that make it much easier to test your code.

Integration Testing

Integration testing (also known as black-box testing) comes after unit testing and it takes a different approach. Integration tests deploy cloud resources and validate their actual behavior but in an ephemeral environment. An ephemeral environment is a short-lived environment that mimics a production environment. It’s often simpler and only includes the first-level dependencies of the code you’re testing.

Some of the behaviors you can verify are:

  • Your project’s code is syntactically well-formed and runs without errors.
  • Your stack’s configuration and secrets work and are interpreted correctly.
  • Your project can be successfully deployed to your cloud provider.

The infrastructure behaves as expected: for example, a health-check endpoint returns a valid HTML document, or a suite of application-level tests succeeds against the public API.

Once the integration tests are finished, you can destroy the ephemeral infrastructure.

A Basic Integration Test

Validating the Shape of Resources

All of the above integration ensures that our program “works” – as in, it does not crash. But what if we want to validate properties of the resulting stack? For example, that certain kinds of resources did (or did not) get provisioned, and that they have certain attributes.

Property Tests

A type of test you may not be familiar with is a property test. Property tests run resource-level assertions while the infrastructure is being deployed. They are there to test your policies and they rely on you having written your policies as code.

In contrast to “black-box” integration testing, policies have access to all input and output values of all cloud resources in the stack. As opposed to unit testing, property tests can evaluate real values returned from the cloud provider instead of the mocked ones.

Use property tests to ensure that your infrastructure complies with your company’s standards. A couple examples are:

  • Checking that you’re using the correct version of the provider’s managed Kubernetes service.
  • Ensuring a service can make an API call to a policy engine to determine whether a request is authorized or not.
  • Ensuring that a resource is provisioned inside a private VPC, rather than the default one.

Security Tests

Too often, security tests are left until the last minute, or code that’s considered “finished” gets thrown over the wall to a security team, who’ve been left out of the entire development process. The phrase “courting disaster” comes to mind when considering this approach. Large companies and governments have all suffered well-publicized data breaches that exposed millions of confidential records.

Security tests should be as much a part of your workflow as any other type of testing. Just as you start testing your code early with unit tests, so should you start testing early to find security problems. If you have a dedicated security team, involve them right away, so they can help you design effective tests. Make sure those tests are included in your CI/CD pipeline.

Just a few of the things you should do are:

  • Strip out all plaintext secrets.
  • Make sure all secrets are encrypted.

Think about adopting services offered by your cloud provider to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.

As with all the other tests we’ve mentioned, security testing should be done as early in the process as possible. If you have a dedicated security team, make sure to involve them immediately. Don’t write your code and then hand off what you consider to be a production-ready code to them. There are a variety of security tests you can incorporate into your development process. Here are a few:

Vulnerability Scanning

  • This kind of testing uses automated software to scan a system against known vulnerability signatures. There are vulnerability scanners on the market that you can use.

Pen Testing (Penetration Tests)

  • This kind of testing simulates an attack from a malicious hacker. This testing involves analysis of a particular system to check for potential vulnerabilities to an external hacking attempt. You might want to check out the The Open Web Application Security Project (OWASP), which is a worldwide non-profit organization focused on improving the security of software. The project has multiple tools to pen-test various software environments and protocols.

Ethical hacking

Try scheduling “game days,” where people in your company deliberately try to hack its systems.

Deployment Testing

By running a program through this integration test framework, you can ensure:

  • Your project’s code is syntactically well formed and runs without errors.
  • Your stack’s configuration and secrets settings work and are interpreted correctly.
  • Your project can be successfully deployed to your cloud provider of choice.
  • Your project can be successfully updated from its starting state to N other states.
  • Your project can be successfully destroyed and removed from your cloud provider.
  • As we will see soon, you can also leverage this framework to perform runtime validation.

Mike Hacker

Written by Mike Hacker, a Native Texan, who lives and works in the Austin/San Antonio, Texas area. You should follow him on Twitter