CodePark/ questions / Contract & end-to-end tests
START PRACTICINGPRACTICE
Backend question · Backend testing › Contract & end-to-end tests

In a system of a dozen services, how many end-to-end tests would you keep, what would they cover, and how do you keep them reliable?

Keep only a few: from a handful to a few dozen. They should cover the journeys that make money, or that would be a disaster if broken. Examples:

  • sign up
  • log in
  • place and pay for an order
  • refund

Lower-level tests cover everything else: service-level integration tests and contract tests. They are faster, and when they fail, they point closer to the cause.

To keep e2e (end-to-end) tests reliable:

  • Run them against an environment deployed the same way as production.
  • Create each test's own data through public APIs, with unique ids. Don't rely on shared seed data.
  • Replace real third-party services with sandboxes or stubs.
  • Wait for conditions with timeouts. Queues and events make the system eventually consistent: data appears a little later, not at once.
  • Capture logs and trace ids when a test fails.

Treat a flaky e2e test (one that passes and fails at random) as a bug. If a scenario keeps failing for infrastructure reasons, move its checks down to a lower test level.

  • End-to-end tests cover only critical business journeys.
  • Lower-level tests and contract tests cover the rest.
  • Each test creates its own data through public APIs.
  • Tests wait for conditions, because the system is eventually consistent.
  • Collect logs and traces on failure, and move unstable checks to a lower level.