Wiki source code of Testing Principles

Last modified by chrisby on 2024/04/01 12:52

version	line-number	content
5.13	1	* Code Coverage: The code coverage is the percentage of classes/methods/lines of code executed during testing. There is class coverage, method coverage and line coverage. Code coverage is generally a useful quality metric and indicator of missing tests, but it is not a guarantee of good tests or that all relevant cases are covered, so take it with a grain of salt.
	2	Aim for 100% code coverage**. Untested code is risky and bug-prone. Although 100% is rarely exactly achieved, it's a worthy goal that encourages thorough testing. The test coverage should be in the high 90s.
2.8	3	Meaningful Tests: **Only add tests that add value, e.g. don't add tests just to achieve 100% code coverage or to test something that has already been tested.
1.1	4	Leverage code coverage tools** that determine code coverage during test execution.
2.2	5	* Failure Strictness: If at least one test fails, it requires the developer's attention. He is not allowed to proceed with other work until that problem is fixed, and all tests must pass without exception.
5.12	6	* Falsifiability = The ability to fail. Tests must be able to fail if the production code does not behave as expected.
1.7	7	** Wrongfully failing tests are easy to detect, because a good developer will pay attention to all failing tests and fix the problems before moving on.
5.12	8	** Wrongfully passing tests lack the ability to fail properly and are hard to detect because everything seems fine, which is why it is so important to focus on preventing them. They do damage by appearing to validate functionality that they do not. A properly executed Test-Driven Development workflow is one of the best ways to avoid these kinds of problems, which are described in [[this article>>doc:Software Engineering.Agile.Extreme Programming.Test-Driven Development.TDD Workflow Example.WebHome]].
5.10	9	* Determinism and Reproducibility: Repeated tests should always produce the same result. The success of tests should never depend on anything other than the production code. Avoid sources of randomness that could cause the test to fail even though the production code is the same as it was when the tests passed in a previous test run. These include random number generators, threads, host-specific infrastructure (operating system, absolute paths, hardware: I/O speed or CPU load), networking over the local network or Internet (since the network or other hosts may sometimes be down), time/timestamps, pre-existing values in the database or files on the host system that are not in the source code.
	10	* Single Action Execution: The entire setup and testing process execution should require a single click or shell command. Otherwise, they will not be used regularly, which reduces the quality assurance of the production code.
5.13	11	* Independence: Tests should be independent and not influenced by other tests; the order in which tests are executed should not matter. Promote test independence by avoiding mutable states. Use constants for shared data between tests where feasible. For mutable shared data that may be modified by tests, setup methods typically initialize instance variables before each test to ensure correct values. The class or application under test can be restored to its original state by calling cleanup methods after each test.
5.17	12	* Self-Containment: For reliability, test results should depend only on the source code to support reproducibility, and not on external factors such as potentially unstable network connections.
2.5	13	* Clean Test Code: Maintain the same high standards for test code as for production code. The only exception is execution speed - tests need to be fast, but not highly optimized, as they typically deal with only small amounts of test data.
5.19	14	* Easy Bug Tracing: Failed tests should clearly indicate the location and reason for the failure. Measures to achieve this could include using only one assertion per test, using error logs where a problem occurs the first time, and using unique exceptions and error messages where appropriate.
5.15	15	** Having only one assertion per test is often a good guideline, but not a strict one, since it's sometimes practical to include multiple assertions within a single test if they are closely related and together validate a specific behavior.
5.7	16	* Test behavior, not implementation: Focus on testing the expected results, not the specific ways in which they are achieved. Tests should not know implementation details. Developers should be free to change production code without breaking tests, as long as the expected results remain consistent. This reduces the tight coupling between test and production code, resulting in more resilient tests that don't need to be constantly updated with every production code change. Therefore, avoid weakening encapsulation by making private things public or using reflection. However, weakening encapsulation is a lesser evil than not testing at all.
5.20	17	* Test requirement fulfillment over specific solutions: Don't limit tests to a specific solution when multiple valid solutions exist. Test for compliance with the solution requirements, not the specific output, as this provides flexibility by accepting any valid solution. For example, in pathfinding problems, minimum travel costs validate a solution, not the particular path chosen. If the solution requirement is complex, it may even be worthwhile to implement a verification algorithm in the test code, since a little test code complexity is the lesser of two evils compared to not testing at all.
5.13	18	* 'Program testing can be used to show the presence of bugs, but never show their absence.' (Edsger W. Dijkstra). Follow good testing practices to avoid a large number of bugs, but be aware that bugs may still occur and be prepared for them.
5.21	19	* Fast: Tests should be small and fast. For example, that there is no need to test million data points, but maybe 5, each covering just one specific use case or border case. The exceptions are load/stress tests, which are specifically designed to send huge amounts of data to an application.
5.22	20	* Specific Assertions: Broad or generic assertions may fail to distinguish between closely related use cases. For example, asserting a thrown exception by catching a generic superclass fails to distinguish between potentially different error cases represented by different exception subclasses. The takeaway from this example is to always assert the lowest type of exception thrown or the specific error message returned.