"We need to build our own testing framework. Standard tools don't cover our specific use cases."
Your Tech Lead has just presented you with a 6-week proposal to develop a custom testing framework for your SaaS application. Solid arguments: greater flexibility, perfect integration with your stack, total control over features.
But before you approve those 6 weeks of development, you should ask yourself one question: does your startup really need a custom framework, or are you falling into the "Not Invented Here" trap?
This is the complete build-vs-buy decision guide for testing frameworks, based on 17 QA implementations at Series A-B startups we've done at onext.
The "Not Invented Here" syndrome in testing
NIH (Not Invented Here) syndrome is the tendency of technical teams to prefer internally developed solutions over proven external ones, even when the external ones are objectively better.
In testing, it shows up like this:
- "Our use case is unique": We believe our application is so special that standard frameworks (Jest, Cypress, Playwright) can't handle it
- "We want total control": We prefer to write our own abstractions on top of existing tools for "greater flexibility"
- "It's faster to do it ourselves": We underestimate the effort of maintaining custom infrastructure
- "Our developers are better": We believe we can build something superior to tools with millions of users and years of maturity
Hard reality: In 14 of the 17 startups we audited, the custom testing framework they built in 4-8 weeks ended up abandoned or replaced by standard tools in less than 12 months. Cost: €30-60k in developer time with no ROI.
Decision framework: 4 criteria for Build vs Buy
Use these 4 criteria to decide whether you should build a custom testing framework or adopt standard tools:
Criterion 1: Is your use case really unique?
Uniqueness test: Answer these questions honestly.
- Are there more than 100 companies in the world doing something similar to what you do? → You're not unique
- Is your stack Node/Python/Java + SQL/NoSQL + REST/GraphQL? → You're not unique
- Do you test web UI, APIs, or service integration? → You're not unique
- Do you need unit, integration, and e2e tests? → You're not unique
If you answered "yes" to any of them, standard tools already cover your use case.
Cases where you ARE unique (build might make sense):
- You test embedded hardware with proprietary protocols
- You test real-time systems with latency constraints <1ms
- You test high-frequency trading algorithms where timing is critical
- You test legacy systems with custom binary protocols from the 90s
Golden rule: If Google, Facebook, Airbnb, and 10,000 other startups solved the same testing problem with open-source tools, you probably can too.
Criterion 2: What's the TCO (Total Cost of Ownership)?
Let's compare the total cost of build vs buy over 3 years:
Scenario A: Build (custom framework)
Year 1:
- Initial development: 6 weeks x 2 senior developers x €65/hour x 40h = €31,200
- Debugging and stabilization: 2 additional weeks = €10,400
- Internal documentation: 1 week = €5,200
- Year 1 total: €46,800
Years 2-3:
- Maintenance (bug fixes, compatibility with new dependency versions): 2 hours/week x 52 weeks x €65/h = €6,760/year
- New features (support for new test types): 3 weeks/year = €15,600/year
- Onboarding new developers: 4 hours per developer x 6 developers x €65/h = €1,560/year
- Years 2-3 total: €23,920/year x 2 years = €47,840
Total cost over 3 years: €94,640
Scenario B: Buy (standard tools: Jest + Playwright + GitHub Actions)
Year 1:
- Initial setup: 1 week x 1 senior developer = €2,600
- Custom configuration (reporters, fixtures, helpers): 1 week = €2,600
- Team training: 4 hours = €260
- Licenses: €0 (open-source tools)
- CI runners (GitHub Actions): €50/month x 12 = €600
- Year 1 total: €6,060
Years 2-3:
- Maintenance: 0 (the community maintains the tools)
- Updates: 2 hours/year x €65/h = €130/year
- CI runners: €600/year
- Onboarding: 0 (official documentation + Stack Overflow)
- Years 2-3 total: €730/year x 2 years = €1,460
Total cost over 3 years: €7,520
Criterion 3: Do you have the internal expertise to maintain it?
Building a testing framework isn't just writing code. It's:
- Designing intuitive APIs so the team adopts them
- Maintaining compatibility with updates to your dependencies (Node, browsers, libraries)
- Debugging edge cases that only appear in CI and not locally
- Documenting exhaustively for onboarding new developers
- Providing internal support when something doesn't work ("why is this test flaky?")
Critical question: Do you have someone on the team who wants to be the "maintainer" of this framework for the next 2-3 years?
If the answer is "no", don't build. You'll end up with a semi-abandoned framework that nobody understands well and nobody wants to touch.
"We built a custom testing framework in 2022. The developer who designed it left the company in 2023. Now nobody knows how it works internally. Every time something breaks, we lose hours debugging. We should have migrated it to Playwright a year ago."
— CTO of a B2B SaaS (Barcelona), 12 developers
Criterion 4: Does building the framework give you a competitive advantage?
The definitive question: does having a better testing framework than your competitors make your clients choose you over them?
Honest answer: No. Your clients don't care whether you use Jest or a custom framework. They care that your product works without bugs.
Your competitive advantage is in:
- Features your competitors don't have
- Superior UX
- Better performance
- Integrations others don't offer
- Smarter pricing
Your testing framework is internal infrastructure. It doesn't generate revenue directly. Invest developer time in what actually differentiates your product.
The Bezos rule applied to testing: "Amazon builds internally only what generates a direct competitive advantage. Everything else is bought or uses open-source." Apply the same rule to your testing framework.
When investing in custom QA DOES make sense
It's not all black or white. There are scenarios where investing in your own QA infrastructure makes sense. But the approach is different from "building a framework from scratch."
The smart option: a QA Center of Excellence (not a custom framework)
Instead of building a framework, you build methodology, processes, and expertise on top of standard tools.
A QA CoE includes:
- A documented testing strategy
- What to test at each level (unit, integration, e2e)
- Target coverage by type of code (business logic: 80%, UI: 50%, utils: 95%)
- When to use mocks vs real dependencies
- Standard tools configured with best practices
- Jest with custom reporters for coverage metrics
- Playwright with the page object model for maintainable e2e tests
- GitHub Actions with matrix testing (multi-browser, multi-OS)
- Visual regression testing with Percy or Chromatic
- Custom helpers and utilities (not a full framework)
- Factories to create consistent test data
- Custom matchers for assertions specific to your domain
- Fixtures for setup/teardown of test environments
- CI/CD optimized for fast testing
- Test parallelization (running 100 tests in 3 minutes instead of 20)
- Smart test selection (running only tests affected by the changeset)
- Flaky test detection and auto-retry
- A culture of quality ownership
- Developers write tests as part of the PR (not a separate QA team)
- Code review includes reviewing the tests
- Visible quality metrics (coverage, flakiness, bug MTTR)
Result: You have world-class testing without maintaining a custom framework.
Real success story: an eCommerce SaaS (Series B, €2.5M ARR)
Initial context (2024):
- Team of 14 developers, 0 QA engineers
- Coverage: 28% (mostly tests written "out of obligation", not conviction)
- E2E tests: 0 (manual testing before every release)
- Production bugs: 12-15 per month
- Manual pre-release testing time: 8 hours per developer
Transformation (8 weeks of QA CoE with onext):
Weeks 1-2: Audit + testing strategy
- We identified 3 critical areas with 0 coverage that caused 70% of production bugs
- We designed a testing pyramid: 70% unit, 20% integration, 10% e2e
- We defined coverage targets by type of code
Weeks 3-4: Tool setup + quick wins
- We configured Jest with custom reporters to visualize coverage by feature
- We implemented Playwright for e2e tests of critical user flows
- We added 187 unit tests in critical areas (checkout and payments business logic)
- Coverage rose from 28% to 54%
Weeks 5-6: CI/CD optimization + flaky test elimination
- We configured GitHub Actions with matrix testing (3 browsers x 2 OS)
- We implemented parallelization: tests dropped from 18 minutes to 4 minutes
- We identified and fixed 12 flaky tests (of 14 total)
Weeks 7-8: Enablement + handoff
- Training across 4 sessions for the whole development team
- Internal documentation: a testing playbook with real examples from the codebase
- We defined 2 internal "testing champions" to evangelize
Results (6 months post-implementation):
- Coverage: 76% (vs 28% initial)
- E2E tests: 42 tests covering 90% of critical user flows
- Production bugs: 3-4 per month (75% reduction)
- Manual pre-release testing time: 0 (everything automated)
- Team confidence: "Now we deploy without fear"
Decision tree: Build vs Buy vs CoE
Use this decision tree to determine your strategy:
Is your use case genuinely unique?
- Yes (embedded hardware, HFT trading, proprietary protocols) → Consider BUILDING a custom framework
- No (web app, mobile app, standard APIs) → Continue the analysis ↓
Do you have >€100k of developer time available to invest over 3 years?
- No → BUY standard tools (Jest, Cypress, Playwright)
- Yes → Continue the analysis ↓
Do you have internal expertise in testing frameworks and someone committed to maintaining it?
- No → BUY standard tools
- Yes → Continue the analysis ↓
Does a better framework give you a direct competitive advantage?
- No → BUY standard tools
- Yes (you're a testing-as-a-service company) → BUILD might make sense
Does your current testing have <60% coverage and frequent production bugs?
- Yes → Implement a QA CENTER OF EXCELLENCE on top of standard tools
- No → Keep your current tools and optimize processes
Common mistakes when deciding Build vs Buy
- Underestimating the long-term maintenance cost
80% of the cost of a custom framework isn't the initial development, it's the maintenance over 3+ years. Don't forget it. - Over-engineering the solution from day 1
If you decide to build, start with a super-simple MVP. Don't build abstractions for use cases that don't exist yet. - Not measuring the ROI of the decision
Define metrics before you start: How much time do you save? How many fewer bugs? How much faster do you onboard developers? If you don't improve those metrics, the decision was wrong. - Ignoring "time to market"
6 weeks building a framework are 6 weeks not shipping features. Is the trade-off worth it? - Not considering the hybrid option (CoE)
It's not build or buy. You can use standard tools + world-class methodology. That's a QA Center of Excellence.
Is your team debating build vs buy for testing?
If your team proposed building a custom testing framework, don't approve it immediately or reject it outright.
Do a 2-3 day technical audit to evaluate:
- Is the use case really unique or does a proven solution already exist?
- What's the real TCO of build vs buy over 3 years?
- Which quality metrics are we trying to improve?
- Would a QA Center of Excellence solve the problem without a custom build?
At onext we implement QA Centers of Excellence at Series A-B startups. In 6-8 weeks, we transform your testing from "coverage <40% with frequent bugs" to "coverage >70% with fast, reliable tests".
Without building custom frameworks. Without maintaining your own infrastructure. Without stopping deliveries.
Written by the technical team at onext, a Spanish applied-AI consultancy. It reflects the team's practice in transforming development teams, cloud, DevSecOps and quality: 12 teams transformed and 0 sprints lost.