Contract testing approaches, compared. Where each one fits, and where Carrick sits among them.

Every approach below answers the same question: does the consumer still agree with the producer? They differ in who writes the agreement down, when it gets checked, what it takes to keep it true, and how much has to exist before the first check runs.

Carrick is the fifth: the contract derived from the code itself, checked on every pull request.

How the derived check works
1.0Five approaches
  1. 1.1

    Consumer-driven contracts

    The consumer writes down what it expects, the provider verifies against it, and a broker records which versions agree.

    The contract is
    Expectations written by the consumer, versioned in a broker
    Checked
    In each side's test suite, and before deploy against what is live
    Stays true while
    Both teams keep writing and updating the tests
    Fits when
    Polyglot systems, deploy gating, message-based interfaces, and anywhere the contract has to say more than the types can.

    Tools that take this approachPact, PactFlow, Spring Cloud Contract

  2. 1.2

    Spec-first with OpenAPI

    A specification is the source of truth. Clients are generated from it and requests can be validated against it.

    The contract is
    A spec file, kept alongside or ahead of the implementation
    Checked
    At build time through generated clients, and at runtime if you validate
    Stays true while
    Someone updates the spec on every change to the API
    Fits when
    Public APIs, external consumers, and teams that already keep their specs true.

    Tools that take this approachSpecmatic, Schemathesis, Dredd, Optic, Microcks

  3. 1.3

    Shared types package

    The producer publishes its types as a package and every consumer depends on it, so the compiler does the checking.

    The contract is
    A published package at a version each consumer chooses
    Checked
    At compile time, in the consumer, against the version it has pinned
    Stays true while
    Every consumer stays on the current version, and someone cuts releases
    Fits when
    A small number of repositories with one owner and a release habit that holds.

    Tools that take this approachA published npm package, or a monorepo with tRPC or ts-rest where the types cross as imports

  4. 1.4

    Integration tests

    Run the services together and exercise the real calls between them.

    The contract is
    Implicit in the running system
    Checked
    After build, in an environment that has to exist and stay representative
    Stays true while
    The environment, the fixtures and the runtime the tests need
    Fits when
    Verifying behaviour rather than shape. The last line of defence, not the first.

    Tools that take this approachTestcontainers, Docker Compose environments, and end-to-end suites

  5. 1.5

    Derived contracts Carrick

    Carrick reads the TypeScript on both sides of every cross-repository call and fails the pull request when the two stop agreeing.

    The contract is
    The code itself, read on every pull request
    Checked
    Statically, in CI, before merge, against the other repository's main branch
    Stays true while
    It is the code, so there is nothing extra to keep true
    Fits when
    TypeScript polyrepos, from two repositories to two hundred. The check covers everything the types say, on every pull request.

    Tools that take this approachCarrick

What each one takes to set up

The steps below are what has to exist before the first check runs. Every approach except one asks for a test suite, a spec, a package or an environment first, and asks for it again each time the API changes.

2.0Setup, step by step

Consumer-driven contracts

  1. Write consumer tests that record every expected interaction
  2. Host a broker and publish the contracts to it
  3. Add provider verification to the provider's CI, with a provider state for each scenario
  4. Wire deploy checks into the pipeline so a version only ships against contracts it satisfies
  5. Keep both sides' tests current as the API changes

Spec-first with OpenAPI

  1. Write the OpenAPI spec, or generate it from annotations
  2. Generate clients and commit or publish them
  3. Add request and response validation against the spec
  4. Add a check that the implementation still matches the spec
  5. Update the spec on every API change

Shared types package

  1. Extract the types into a package
  2. Publish it to a registry, with versioning
  3. Bump every consumer on each change
  4. Add a check that no consumer is pinned to a stale version

Integration tests

  1. Stand up an environment with every service in it
  2. Seed fixtures and test data
  3. Write the tests that exercise the cross-service calls
  4. Keep the environment representative as services change

Derived contractsCarrick

  1. Install the GitHub App on the repositories
  2. Add the Action to the workflow

Every pull request after that is checked.

Where Carrick fits

Carrick is the one approach here where the contract keeps itself. It is the TypeScript on both sides of every cross-repository call, read on each pull request, so it is true for exactly as long as the code is.

There is no test suite to write, no broker to host and no spec to keep in step. It fits TypeScript polyrepos of any size and installs as a single GitHub Action. In a system that also has services in other languages, or that gates deploys against what is live in production, Carrick runs beside a written contract and covers every TypeScript call.

3.0The derived checkHow it works
4.0FAQ

Common questions

Different questions. Contract testing asks whether two services still agree on the shape of what passes between them, and it can answer before anything is deployed. Integration testing asks whether the services behave correctly when they actually run together, and it needs an environment to answer in. Most teams that have both use the contract check to catch the cheap, frequent failures early and keep the integration suite small.

Yes, and for a polyglot system that is the sensible arrangement. Carrick covers the TypeScript-to-TypeScript calls on every pull request with nothing to author, and the written contract covers the services it cannot see and the guarantees a type cannot express, such as a field's format or which version is live in production.

Catch it in the pull request, not the incident channel. Carrick reports where a producer and a consumer no longer agree, on every pull request, across every repository you connect.

Get started now Apply to get 3 months free Free for up to 10 repositories