Automating Tests: Deciding What to Automate, What Not To…

Short answer: Test automation is writing code that proves other code works, running it on every commit, and treating a red run as a blocker. The practice earns its keep when the suite catches regressions before humans do, runs in minutes not hours, and does not need babying every sprint. In 2026 the breakdown is still the pyramid — unit tests at the base, integration in the middle, a thin end-to-end layer on top — and the common failure modes are still flaky tests papered over with retry flags, suites that grow without an owner, and parallel runners split by file count instead of observed runtime.

Test automation is about verifying code works by writing code that does this verification. Running this verification code on every commit and treating a red build as a blocker rather than a suggestion. The practice has been around since the 1990s. The pyramid diagram has made the rounds since 2009. The tools have changed RSpec, Minitest, Jest, Vitest, pytest, Playwright, Cypress but the shape of the problem has not. Bugs are still shipped by teams that could have been prevented by a thirty line unit test. There are still teams running 400-spec suites that take twelve minutes to tell them that a timezone changed on one runner.

This post is a field guide, not a lesson. We will explore what test automation offers, the three layers worth automating, the math of return on investment, and four failure modes that stealthily consume teams. If you have been writing automated tests for 10 years, proceed straight to failure-modes. If you're taking over a CI pipeline for the first time, coming from a QA-only environment, read this whole thing.

Ruby.ci is a self-hosted CI/CD that has plug-and-play runners for Jenkins, GitHub Actions, Buildkite, and GitLab. We are here mentioning it. The rest of the piece is at the category level; the practice travels across tools.

What test automation actually is

The standard definition of CI/CD is software executing test cases autonomously and reporting pass/fail. Yes, that's right but useless. A more sincere working definition.

Test restoration is a two-piece contract. The initial component consists of actionable statements regarding the expected behavior of the system. The second part is some mechanism (usually CI) that runs those assertions on every change and doesn’t merge changes that break them. Both elements need to function. Without a CI gate, a comprehensive test suite merely serves as documentation. A CI gate with a shallow test suite is just an act.

The practice's goal is to reduce the turnaround time on identifying regressions. Without automated tests, a change that we make on Tuesday afternoon shows up as a bug in production on Friday evening, just as everyone is leaving. If you had run your tests on the previous push, you would have a red check on the pull request four minutes later. Automation Is a Vehicle The length of feedback loop is actual deliverable.

It is important to note that this framing is the only one which can help one decide on what to automate. Anything whose regression you’d want to catch in minutes instead of weeks is a candidate. If you can accept the regression being caught during a monthly QA cycle, they are not a CI issue. To see a longer riff on this, check out our software testing pyramid piece – which adapts the same idea to a diagram.

The three layers that earn their keep

Understanding Mike Cohn's Pyramid Made Easy.

• Unit tests. One function or class, no external dependencies, run in milliseconds. 70-80% of your suite by count. The cheapest to write, cheapest to run, and the layer that catches the most bugs per dollar spent.
• Integration tests. Multiple components talking — usually a service hitting a real database or a message queue. Hundreds of these, not thousands. 15-20% of the suite. More expensive but catch bugs the unit layer cannot see.
• End-to-end tests. The whole system driven through a browser or an API client. Dozens, not hundreds. 5-10% of the suite. The most expensive to run and the most flaky. Use sparingly.

The system has varying ratios. A stateful backend will have more integration tests than a frontend-heavy app. The stateful backend has more integration tests than a frontend-heavy app. But the shape holds. When the pyramid is upside down (having more E2E and less unit tests), it slows down the suite, creates flakiness, and can be demoralising. Inside a suite, the architecture is a design choice and not a vibe.

For Ruby on Rails application development, it just has layers The unit layer of the application contains model tests. Request specifications and controller tests are located in the integration layer. Either Capybara or Playwright feature specs are the core of your application. The LINKPLACEHOLDER4END piece takes a closer look at the lowest layer.

ROI math that actually holds

"The common pitch for test automation is that you'd move faster." That's accurate, but unprovable. The cost per catch refers to the savings that a test suite generates due to a bug being fixed before shipping (versus after) and the cost of maintaining and running the test suite.

The inputs for the math are four in number.

1. Bug escape rate. How many bugs reach production per month without the suite vs with it. If you are not tracking this, start — one spreadsheet column counts.
2. Cost per escape. Engineering time to diagnose, patch, deploy a hotfix. Figure 4-8 hours of senior-engineer time per customer-facing bug. For regulated industries, add compliance surface.
3. Suite maintenance cost. Engineering hours per month spent fixing flakes, updating fixtures, triaging failures. A healthy suite is one to two engineer-days per month for every 500 tests. A sick suite is a half-time job.
4. Suite runtime cost. CI minutes consumed per week times the vendor's rate. Hosted CI like GitHub Actions bills per minute; self-hosted CI like Ruby.ci bills by compute. Compare line items.

The math is straightforward. If a suite identifies 20 escapes in a month at 6 hours each, that’s a saving of 120 engineer-hours a month. If maintenance and runtime takes forty engineer-hours a month, net eighty hours is a handsome return. If it takes 160 hours to maintain, the suite is a tax, not a benefit.

This math is almost never done. Those who do make a stop of writing E2E tests for each form field within a week.

What to automate first

When starting a suite from scratch, the order is important. It can be tempting to default to writing unit tests for the most topical module. It’s worth the effort to write a regression safety net for the three things you’ll be most embarrassed about if they break.

• Billing / payments. A charge that goes to the wrong card gets a lawyer involved. Automate this first.
• Auth. Login that silently returns someone else's session is a company-ending bug. Automate this first.
• Data integrity. Writes that drop records, updates that race, migrations that corrupt. Automate this first.

The rest, including form validation, paginated list, the sort order on the admin dashboard, can wait for a sprint. The first fifty tests written, in general, should answer what will, if it breaks, cost the company. The following 500 respond what feels brittle.

For Rails teams, this translates to having request specs on billing endpoints, controller specs on sessions, and model specs on anything that involves a transaction. The baseline is Bundler and RSpec. Something's up if your Gemfile doesn’t contain a test gem that costs more than a line.

Parallel execution, done honestly

A suite of 400 specs takes twelve minutes on a single runner but under two minutes on ten runners, assuming you split it correctly. Many people mistakenly set their runners based on file count; for instance, if 400 files exist, 10 runners will be set with 40 files per runner. When 399 specs take a few seconds to run, but one runs for eight minutes, it takes that eight-minute one that sets the pipeline’s wall clock. The other nine runners complete their tasks within a minute but remain unused afterward.

Runtime-balanced splits distribute specs according to actual runtime data. The runner responsible for the slow spec executes fewer overall specs. The differences in finish times between the runners is not large. Knapsack Pro achieves this for Ruby suites. Certain CI providers include this as a built-in feature. By default, Ruby.ci's parallelized runner balances observed runtime. If you are partitioning based on the number of files and achieving less than a 1.5 times performance boost due to parallelism, the problem lies with the division and not the suite.

The second most common mistake is parallelising a non-parallel-safe suite. A database that multiple apps access, hardcoded ports, and code with race conditions on cached objects. Make those corrections prior to coordinating. If your test suite fails 5% of the time, you’re probably wasting time chasing false positives. Such failures are best fixed on a deeper level.

The four failure modes that eat teams

Test suites that are automated rarely die due as a result of neglect. They die due to four slow-moving diseases.

Flakes papered over with retry flags

A suite with a 3 percent flake rate and a --retry 3 on each job is a 99.9 percent green suite on paper. Essentially, your CI costs have doubled, you have concealed an actual bug, and you have shown the team that red builds are to be "retried". Putting flake tests in a quarantine job is a smart strategy. Such tests do not block merge requests and the results are displayed for the liable team to resolve. Removing the test conceals the defect. When you retry, you're covering up the signal. Isolation safeguards both.

No single owner

The configuration that we audit most frequently is an RSpec config, a .github/workflows/ci.yml, and a spec_helper.rb that six engineers have modified in the last year. No one can describe the whole setup end to end. New flaky jobs are added, and unutilised jobs are kept. The file accumulates YAML similar to how bicycles fill up a garage. Choose 1 owner not a committee. The owner evaluates any change to the pipeline. Changes made to the pipeline are changes made to code.

Suite grown without pruning

A suite with 2000 specs is worse than one with 600 specs if the former has 60% feature-coverage and the latter has 85% coverage on the right things. Quality does not depend on volume. Eliminate any tests that offer redundant coverage, test implementation details rather than behaviour, or simply exist as a means for someone to add a test to “get the PR in”. A test should fail for only one reason and you should be able to name that reason in one sentence.

Integration tests pretending to be unit tests

A 400ms “unit” test that hits a real database is an integration test in disguise. The disguise is significant because the suite's performance budget is calibrated for the unit layer - ten thousand ten millisecond tests and not a thousand four hundred millisecond tests. Once a quarter, check the live runtime distribution. The integration tests are usually mislabelled the slow outliers.

Tooling in 2026

An Accurate Assessment of the Stack

The sister articles on alternatives to Jest and alternatives to Vitest explore the JS side in detail.

To enhance your CI tool, try to use the same reference pattern: fetch, install, cache lockfile, run suite, upload results. The YAML is different, not the actual practice. Refer to how to continuous integration for the pipeline design.

What test automation does not do

An honest primer identifies the limits. Automated tests do not.

• Replace code review. A green suite means no test regressed. It does not mean the design is right or the migration is safe.
• Replace exploratory QA. Humans find edge cases that neither the tests nor the test authors imagined.
• Catch bugs that are not represented in tests. A race condition that was never asserted will not be caught by a run of the suite.
• Make a bad codebase good. The suite is a mirror. If the code under test is tangled, the suite will be tangled too.

Many assume the change is correct when in reality, they are wrong. When a build is green it means that nothing already known to tests has been broken by this change. The minimum, not the maximum.

When to skip test automation

Not every project is an enormous 500-spec suite. Examples of situations where investing in more tests is not feasible.

• A one-off script that will run three times and be deleted. Write it, review it, run it, move on.
• A prototype where the goal is to learn whether the design works. Tests freeze the design. If you have not finished learning yet, do not freeze.
• A legacy module scheduled for replacement in the next quarter. Characterisation tests for the data in and out, nothing more.

On the other hand, it is advisable to test the module if it affects billing, authorisation, data integrity or if a workflow regulated by law, even if the road-map says it dies in six months. You will realize that the roadmap you envisioned was inaccurate during the sixth month.

A short glossary

• Test suite — the full set of automated tests for a project.
• Test case / spec / test — one assertion. Interchangeable in casual use; "spec" is Ruby-flavoured.
• Fixture — pre-built data used by a test. Minimise.
• Mock / stub / spy — different ways to replace a real dependency with a controlled one. Mocking is the broadest term.
• Coverage — percentage of code lines (or branches) exercised by the suite. Useful signal, bad target.
• Flake — a test that passes and fails on the same code without the code changing. A bug, not a mood.
• Regression — a bug introduced by a change that previously was not there.

If you want to learn about acceptance, integration and unit breakdown, read the unit testing vs integration testing article and acceptance testing explainer.

Frequently asked

What is test automation?

Test automation refers to the practice of writing code to prove other code works, running this on every commit, and treating the red build as a blocker. Essentially, a test suite augmented by a CI gate. Both sides need to work for it to earn its keep.

What are the benefits of test automation?

Less time spent debugging regressions (minutes instead of weeks), lower bug production, less human resources spent on repeating verifications, and increased confidence in refactors. If you track bug escape rate and suite maintenance cost, then the ROI is measurable.

What should I automate first?

The three things you would most hate to see break are billing or payments, auth and data integrity. Anything else, such as validating forms, paginating, and sorting the admin dashboard, can wait one sprint.

What is the test pyramid?

The pyramid encourages teams to have most tests as fast unit tests, fewer tests as integration tests and a thin set of end-to-end tests at the top. A reversal of this approach, where end-to-end testing is carried out more than unit testing, generates slow and flaky suites that are not trusted by anyone.

Why do my tests fail only in CI, not on my laptop?

The most common reasons for flaky builds are a stale cache state on the runner, timezone or locale differences between CI and your development machine, parallel execution exposing a race condition, and/or an environment variable set differently on CI. Four common causes are specified above. Most of these issues can be resolved with ephemeral runners.

How do I measure test automation ROI?

Monitor the following four figures: the bug escape rate, which proposes the quantity of bugs which slip through to production each month; the cost of escape, which proposes the engineering hours taken in diagnosing and fixing; the suite maintenance cost, which proposes the hours taken to maintain the suite each month; and the suite runtime cost, which proposes CI minutes times vendor rate. The net savings will tell you if the suite pays for itself.

What is a healthy flake rate?

Less than 1% If it exceeds 2 percent, the team won't rely on the red builds. Once the number goes over 5%, build suite starts to become a liability. Merges happen on retry, not on real green. If your flake rate exceeds 2%, avoid adding tests until it decreases.

Still stuck

If your suite is under three minutes long and under a 1% flake rate, and the invoice isn’t making Finance ask questions, keep it. The practice doesn’t require a new tool.

If one of those three is wrong a twelve-minute cold Rails build, a flake rate above two per cent, or a hosted-CI bill you cannot explain the fastest fix will usually not be a new test framework. It is one more runner. Ruby.ci integrates seamlessly with Jenkins, GitHub Actions, Buildkite, and GitLab without requiring changes to your app or config files. Set one up on a mid-tier box to gauge. These are the terms of our deal with the client.