focus for this week: aspects of neuroscience
(popular pages, total: 2,363, >2 mio views)
Figures about test effort costs
Contents |
We all know the graphics, where you can see,
that the "longer a defect remains undetected, the more expensive it becomes to correct"[1]
(These are mostly referrenced with Barry Boehm,
but I guess everybody here knows from his/her experience that this is true)
figure with friendly permission from Steve McConell[2]
Please keep in mind, that the above are average values -
there may be also examples, where the above may not be true:
e.g.
an unexperienced developer A:
while searching for a fault which was introduced in implementation,
A perhaps can not fix the fault correct (so, that other cycles are needed later on) or does need much time to find the solution or the anomaly itself or perhaps can not solve it and must involve another developer.
While perhaps B can recognize a fault introduced in the requirements while testing (shortly before release): e.g. the fault is in a function, which is not so important or has no relation to other features, then fixing the spec and of course the code does probably not cost much.
In the following there will be listed some numbers from literature and other public sources.
Please consider:
- that these numbers are generally speaking (this may vary due to project-specific or other conditions)
- more test-effort does not always imply better quality
- the listed figures are perhaps not comparable always with each other, but the goal is, that you have some hard facts, which you can use, if you need them as a lead (for different situations: e.g. to compare with your actual test effort, get some more time for tests,...).
I will keep this updated, so that this list gets bigger and bigger....
If you want to help, send me a link and a figure.
test effort in relation to total costs
| source | test effort | |
| Martin, J., McClure, C. (1983)[3] | 15% | of relative costs |
| Craig, R. D. und Jaskiel, S. P. (2002, S.9) | 20-50% | of software development budget |
| Hetzel, B. (1988) in: Burnstein, I. (2003, S.245) | 25% | direct testing costs of total development costs |
| Koomen, T. und Pol, M. (1999, S.1) | 25-50% | of total project costs |
| Spillner, A. und Linz, T. (2005, S.2) | 25-50% | of developping time and developing costs |
| Fewster, M. und Graham, D. (1999, S.458) | 30% | of total project costs |
| doIT online (2006) | 33% | of software development costs, SAP (when considering 18 months of developping time) |
| Illes T., A, H., et al. (2005, S.4) | 30-40% | of software development costs |
| van Veenendaal, E. und Swinkels, R. (2005, S.341) |
30-40% | of total project costs |
| Sogeti Deutschland GmbH (2003, S.1) | 30-40% | of total budget |
| ASQF-Fachgruppe Software-Test (2006) | 30-60% | of total development costs |
| Link, J. (2005, S.337) | 40% | of total development costs (author says: rule of thumb probably for new projects) |
| Spillner, A. und Linz, T. (2005, S.181) |
50% | of total development costs (the test effort over all test levels) in typical application development projects |
| doIT online (2006) | 50% | of software development costs, Bill Gates |
| Myers, G. J. (2001, S.vii) | >50% | of total project costs |
| Kit, E. (1995, S.7) | >50% | of the development effort (is frequently spent on testing) |
| Beizer, B. (1990, S. xiii) | >50% | generally speaking: labor that is expended in testing |
| Laprie, J.-C. (1994, p.408)[4] | 50-75% | of total development costs (for verification + validation activities: "ordinary" systems: 50%; safety-critical software: <=75%) |
| Musson, R. in: Link, J. (2005, S.338) | 66% | of total development costs (for integration and system testing) |
| Ubar, R. (1994, p.151)[5] | >70% | of production costs (for hardware, e.g. VLSI, ASICs, digital systems) |
| Kit, E. (1995, S.20) | 40%-70% | of initial software development time + resources is devoted to error detection + removal |
test effort in some phases of the test process
| what experience from test point analysis from TMAP approach shows |
of total test effort (when developing new software) | of total test effort (during maintenance testing) |
|
| test preparation | 10% | ||
| test specification | 40% | ||
| test execution | 45% | 40% | >40% |
| test completion | 5% | ||
| source | van Veenendaal, E. (2005, S. 129) |
van Veenendaal, E. (2005, S. 371) | van Veenendaal, E. (2005, S. 371) |
| relative efforts for development activities | Link, J. (2005, S.337, rounded) |
| analysis | 9% |
| implementing without tests | 16% |
| code inspections | 1% |
| unit test | 8% |
| integration test | 17% |
| systemtest | 49% |
| Phases | relative costs (J. Martin, C. McClure(1983)[3] |
| requirements analysis | 3% |
| specification | 3% |
| design | 5% |
| coding | 7% |
| testing | 15% |
| operations + maintenance | 67% |
Gilb in:Spillner, A. und Linz, T. (2005, S.79)
10-15% of development budget (for costs through reviews)
costs in aspect with maintenance
| Link, J. (2005) |
>40% test effort | of total costs (author says: this is rule of thumb for revisions and maintenance projects) |
| Trifu, M. (2006, S.7) |
67-80% maintenance costs | of total costs |
| Martin, J., McClure, C. (1983)[6] | 67% maintenance costs | of total costs of software |
|
activity
|
maintenance budget is spent on [6]: |
| fixing errors | 20% |
| adapting the program, so that it works with new HW or with new co-resident SW | 25% |
| fixing the documentation | 6% |
| performance improvements | 4% |
| making changes (enhancements) requested by users | 42% |
todo
update:
- new figures
- general goal of the page
- Why are there not higher values?
- How did the authors get the "value" exactly?
- Are the figures really that important? perception/"reality" over time?...
history
This is the page of my old site (2006) with figures about test effort costs
sources
- ↑ Steve McConnell (1996): Software Quality at Top Speed. Software Development, August 1996
- ↑ Steve McConnell (2004): Code Complete, 2nd Edition, Microsoft Press, Redmond, Washington. ISBN 0735619670
- ↑ 3.0 3.1 J. Martin, C. McClure(1983): Software Maintenance: The Problem and Its Solutions. Englewood Cliffs, NJ: Prentice Hall
in:
Cem Kaner, Jack Falk, Hung Q. Nguyen (1999): Testing Computer Software, 2nd Edition (Paperback), John Wiley & Sons, Inc. - ISBN 0471358460, page 31 - ↑ Laprie, J.-C.: Dependability: The Challenge for the Future of Computing and Communication Technologies in: Dependable Computing - EDCC-1 First European Dependable Computing Conference, Berlin, Germany, October 4-6, 1994. Proceedings. Springer Verlag. ISBN 978-3-540-58426-1
- ↑ Ubar, R.: Test Generation for Digital Systems Based on Alternative Graphs in: Dependable Computing - EDCC-1 First European Dependable Computing Conference, Berlin, Germany, October 4-6, 1994. Proceedings. Springer Verlag. ISBN 978-3-540-58426-1
- ↑ 6.0 6.1 J. Martin, C. McClure(1983): Software Maintenance: The Problem and Its Solutions. Englewood Cliffs, NJ: Prentice Hall
in:
Cem Kaner, Jack Falk, Hung Q. Nguyen (1999): Testing Computer Software, 2nd Edition (Paperback), John Wiley & Sons, Inc. - ISBN 0471358460, page 57
