Test automation frameworks
The types, every major tool, how they work, how a good framework is built, and which one to choose when.
What "framework" means
A test automation framework is the set of tools, code structure and rules that make tests quick to write, reliable to run and cheap to maintain. People use the word for two separate things, and mixing them up causes many bad decisions.
The design type
How you organize your test code and data. Modular, data-driven and BDD are design types. You choose this yourself.
The tool
What actually drives the app and runs the tests. Playwright, Selenium and Appium are tools. Any tool can support several design types.
So "Should we use Playwright or data-driven?" is not a real choice. A common good answer is both: Playwright as the tool, with a hybrid design that uses page objects and data-driven tests.
The design types
Here is the same scenario in each style: applying for a Gold card. Select a type to see how it works and when it fits.
How the tools work
The biggest difference between tools is how test code reaches the app. It explains most of their strengths and limits.
Selenium
- Test code
- WebDriver protocol HTTP
- Browser driver
- Browser
Each command is a separate request to a driver such as ChromeDriver, which controls the browser. It is a W3C standard, so it reaches real browsers widely. Selenium 4 also adds WebDriver BiDi, a two-way WebSocket connection for things like network and console events. You usually build the waiting and structure yourself or with libraries.
Playwright
- Test code
- Persistent connection WebSocket
- Browser engine
One long-lived connection carries commands and events both ways: the Chrome DevTools Protocol for Chromium, and similar protocols for its own Firefox and WebKit builds. Because the tool sees page events directly, it can auto-wait and intercept network traffic. Isolated browser contexts make parallel tests cheap.
Cypress
- Test code, inside the browser
- Your app, same browser
- Node process tasks, network proxy
Tests run in the browser next to the app, giving direct access and an excellent debugging experience. That design also sets its limits: test code is JavaScript or TypeScript only, it controls one browser at a time, and crossing origins needs cy.origin.
Appium
- Test code
- Appium server WebDriver
- Driver XCUITest, UiAutomator2
- iOS or Android app
Appium applies the WebDriver idea to mobile. Its server passes commands to a platform driver, which uses Apple's or Google's own automation tools. Tests can stay in your preferred language and share a structure across platforms, but they run slower than native tools like Espresso or XCUITest.
Maestro
- YAML flow
- Maestro CLI
- Device automation
- iOS or Android app
Flows are read and run step by step, so you can edit and re-run without compiling. Each command waits for the screen to settle and retries, which removes most timing code. The trade-off is less programming power than a code-based framework.
Native mobile frameworks
- Test code, built with the app
- Espresso or XCUITest
- App on the device
These are Google's and Apple's own tools, run as part of the app's build. They are fast and reliable, and suit developers, but you write separate tests for each platform.
BDD frameworks
- Gherkin scenario
- Step definition
- Page object
- Driver tool
- App
Cucumber doesn't drive the app itself. It connects plain-language steps to your code, which uses a tool such as Playwright or Selenium.
API test frameworks
- Test code
- HTTP request
- Your service
No browser is involved. Tools such as Rest Assured, or Playwright's request fixture, send a request and check the response. These tests are far faster and steadier than UI tests, so they should carry much of your coverage.
Compare the tools
On a narrow screen, scroll the table sideways.
| Tool | Best for | Languages | Runs on | Main limit |
|---|---|---|---|---|
| Playwright | Modern web apps, with built-in runner, tracing and parallelism | TypeScript, JavaScript, Python, Java, C# | Chromium, Firefox, WebKit; mobile is emulated | No native mobile app testing; WebKit is not the same as real Safari |
| Selenium | Wide language and real-browser coverage, large existing suites, Grid | Java, Python, C#, JavaScript, Ruby | Chrome, Firefox, Edge, Safari | You assemble more yourself: runner, reporting, waiting strategy |
| Cypress | Front-end teams who want a fast, visual feedback loop | JavaScript, TypeScript | Chrome-family browsers and Firefox; WebKit is experimental | One browser at a time; multiple tabs need a plugin |
| WebdriverIO | JavaScript teams covering both web and mobile | JavaScript, TypeScript | Browsers, plus mobile through Appium | Smaller community than Selenium or Playwright |
| Maestro | Quick mobile user flows, with little setup | YAML, with optional JavaScript | iOS, Android, and web | Less suited to complex logic |
| Appium | Native, hybrid and mobile web apps on real devices | Most major languages through client libraries | iOS, Android | Slower and more setup than native tools |
Tools that work alongside these
A framework usually combines several parts. Runners: JUnit, TestNG, pytest, Playwright Test. BDD: Cucumber, Behave. Keyword-driven: Robot Framework.
Other kinds of testing
API: Rest Assured, Postman, Playwright's request fixture. Performance: k6, JMeter. Native mobile: Espresso on Android, XCUITest on iOS.
All frameworks, and when to choose them
Select a category. Each entry says how the tool works, when to choose it, and when to look elsewhere.
Which to choose, by situation
Find the row closest to your team. These are sensible starting points, so try them on your own app first.
| Your situation | Start with |
|---|
How a framework is designed
Good frameworks are layered. Each layer knows only about the one below it, so a change in the UI or an environment touches one place instead of hundreds of tests.
- TestsWhat should be true, in business terms. No selectors and no raw URLs.
- FlowsReusable journeys that cross pages, like "complete an application".
- Page and API objectsHow to use each screen or endpoint. Locators live only here.
- Fixtures and test dataSetup, cleanup, logged-in users, and data builders.
- Core utilitiesShared helpers for dates, API clients, logging and the like.
- ConfigEnvironments, browsers, timeouts, retries and secrets from outside the code.
- Reporting and CIReports, traces, screenshots, parallel runs and pipeline triggers.
tests/
apply/gold-card.spec.ts # tests
smoke/ # fast checks for each pull request
flows/
application-flow.ts # multi-page journeys
pages/
cards-page.ts # locators and page actions
api/
applications-client.ts # API setup and checks
fixtures/
index.ts # test.extend with shared setup
data/
applicant-builder.ts # typed test data
utils/
config/
environments.ts
playwright.config.ts # browsers, retries, reporters
.github/workflows/tests.yml # CI pipelineDesign patterns you'll meet
- Page Object ModelA class per page or component holds locators and actions. The default for UI tests.
- ScreenplayTests describe actors performing tasks, instead of pages. It scales well for large suites, though it's more to learn. Serenity/JS and Serenity BDD support it.
- FixturesReusable setup and cleanup that tests request by name.
- Builder and factoryCreate valid test data, overriding only the fields a test cares about.
- Arrange, act, assertSet up, do one thing, then check the result. It keeps each test easy to read.
Rules that keep it healthy
- Every test stands alone.It creates its own data and passes when run by itself, in any order.
- Test at the lowest useful level.Check business rules through APIs. Save UI tests for key user journeys.
- Set up through the API, verify through the UI.Only click through the part you are actually testing.
- Wait for conditions, never for time.Fixed sleeps are slow and still flaky.
- Make failures easy to read.Clear test names, traces and screenshots should explain a failure without a re-run.
- Treat test code as production code.Use code review, linting and type checks, and delete tests that stop earning their place.
Signs of a struggling framework
Selectors appear in test files. One UI change breaks dozens of tests. Tests fail unless they run in a certain order. The team re-runs failures until they pass. Only one person understands the core code.
How frameworks usually grow
Start with a few critical journeys and page objects. Add fixtures and data builders when setup starts repeating. Add parallel runs and CI early. Build shared layers only once two or three tests need them.
Which should you choose?
Answer four questions for a starting recommendation. It is a good first shortlist, so run a small trial on your real app before you commit.
Test yourself
What would you do in each situation?
Put it into practice
The practice paths have hands-on topics for the ideas in this guide, and the Test Strategy guide covers what to test and when.