Quality Assurance (Qualitätssicherung) — Final Exam

Why quality assurance matters and where errors come from

Software is never error-free (Murphy's Law: "anything that can go wrong will go wrong"); modern software has millions of lines and therefore millions of chances to go wrong — studies suggest 15–50 errors hide per 1000 lines of code. Famous failures: Therac-25 (the classic "test failure" example), a 2004 Austrian software fault that paralysed phones, and a 2007 Skype update causing a worldwide outage.

Automatic tests are widely used in practice: among the 100 most popular Java GitHub projects there are 150,000+ unit tests; SQLite ships 644× as much test code as production code; Apache Commons, OkHttp, JodaTime, Apache HttpCore average over 82% coverage. If old functionality keeps breaking after changes, (automatic) tests are a key step to improve the situation.

• Faulty requirements documentation: wrongly defined, missing, incomplete or unneeded requirements.
• Misunderstandings from broken customer–developer communication (e.g. misinterpreting requirements).
• Logical design errors: architects/developers formulate faulty or insufficient requirements/architectures.
• Coding errors: misinterpreting the design document, language errors, faulty test-data selection.
• Non-compliance with standardized coding/documentation guidelines, leading to misunderstandings and wrong assumptions about code.

Software quality factors

Quality factors let us make statements about a software's quality:

• Functionality (Funktionalität): presence of functions with defined properties.
• Reliability (Zuverlässigkeit): maintaining a performance level under given conditions over time.
• Usability (Benutzbarkeit): the effort needed for users to use the software.
• Efficiency (Effizienz): ratio of performance level to resources used.
• Changeability (Änderbarkeit): effort needed to make given changes.
• Portability (Übertragbarkeit): how easily the software migrates to another environment.

Costs of quality assurance and cost optimization

Total costs combine production costs and quality costs. The cost of fixing an error — and thus the quality cost — rises exponentially with the error's latency (the gap between when an error is made and when it is fixed). A key goal is to avoid errors early and prevent their follow-up costs.

Cost optimization is the point where the ratio between error costs, error-avoidance costs and quality is optimal — where the cost of avoiding/checking errors equals the cost of fixing them, and total costs are lowest. (Quality-cost question 2.1: testing effort itself costs — creating/maintaining manual or automatic tests — so even good quality has a price.)

Static QA and static analysis

Static QA covers all analytical activities that check a test object without executing it (e.g. code reviews); its advantage is that test objects can be checked very early, before they are runnable. Static analysis ensures the "internal quality" (software structure and design): it checks individual parts long before they are assembled into a complex running system.

• Structure analysis: determines and guarantees internal quality across the whole process, based on dependency graphs and quality metrics.
• Error-pattern analysis: checks source against typical error patterns.
• Static analysis tools: check compliance with development guidelines.

Software metrics

Software metrics measure software quality. Process metrics measure process properties (quantitative process data); product metrics measure the software (or parts) regardless of how it came to be.

• Size metrics: measure module size and help decide when to split oversized modules — LOC (Lines of Code) or the more meaningful NOS (Number of Statements).
• Halstead metric: assumes a program consists of operators and operands, counted via formulas.
• Logical structure metrics: measure module structure such as number of paths or loop-nesting depth — best-known is the McCabe metric (cyclomatic complexity, the most used).
• OO metrics measure object relationships/structure (how good is the OO design?), driven by the Block-3 design principles: CBO (coupling between objects), DIT (depth of inheritance tree), RFC (response for a class), WMC (weighted methods per class), LCOM (lack of cohesion in methods, ideally 1).

Reviews: purpose, roles, and process

A review is a formally organized meeting of people to check a product part's content or form against given criteria/checklists — a qualitative assessment of products and processes, not of the author. Its effectiveness depends on correct conduct.

• Advantages: applicable to all development results (requirements, designs, source code); can run very early before executable programs exist; shortens error latency, reducing fix costs (offsetting the review effort); relativizes developer self-assessment and fosters knowledge transfer.
• Disadvantage: the author can become 'the accused', making reviews unpleasant or team-damaging (e.g. dwelling on single errors). Solution: build an open, critical but fair review culture.
• Roles: Manager (commissioned the object, responsible for release); Moderator/Review leader (keeper of process — plans, organizes, leads, ensures an open atmosphere); Recorder/Scribe (preserver of knowledge — writes the review report); Author (creator, attends to clarify but not justify); Reviewer/Inspector (examines and reports findings; cannot also be the author); Reader (keeper of focus and pace).
• Process phases: Planning, Initialization, Preparation (the core: reviewers search for defects), Session (defects communicated and documented, not corrected), Rework (author corrects), Third Hour (optional discussion), Analysis (optional systematic evaluation/process improvement). Preparation/Session and Rework are necessary; Third Hour and Analysis are optional.

Review types and code reviews

• With customer: Software Requirements Review (SRR, after requirements before design), Preliminary Design Review (PDR, in design phase), Critical Design Review (CDR, for especially critical components before implementation), In-Process Review (IPR, shows progress/prototypes/test cases — important for large projects to avoid developing past customer wishes).
• Without customer: Management Review (MR, formal project-status assessment); Inspection (formal review) and Code Walkthrough (weakened review) — strongly formalized types aimed at finding defects, the most common/important/effective review method; Technical Review (checks a concrete part against specs/standards).
• Round-Robin Review: an alternative where reviewers search for positive arguments and try to convince colleagues of the object's quality.
• Code reviews (a classic you will give/receive): analyse code in advance, identify wrong decisions and their type (syntax, runtime, logic), explain/justify findings face-to-face, give improvement suggestions and how to avoid the issue in future — treat them as reasoned tips, not criticism. Reviews should be combined with technical QA (e.g. unit tests).

Goals of testing and what makes a good test

Of three common definitions — (1) testing demonstrates a program does what it should, (2) testing runs a program intending to find errors, (3) the IEEE 'operate under specified conditions, observe/record results, evaluate' — the error-finding view (2) and the neutral evaluation view (3) are preferred; view (1) only seeks confirmation.

A good test has a high probability of finding errors; a bad test rarely does. You can "test the tests": e.g. mutation testing deliberately injects faults to check whether the tests catch them, or measure how many real defects the tests detect (better than coverage alone).

• Goal: establish the correctness/completeness of functions and find errors as early as possible.
• A good test identifies new, unknown errors; the priority is to find and document errors reproducibly so they can be fixed (reproducibility is secondary).
• The number of errors is unknown, so you can never prove all errors are found — therefore cover as much functionality/behavior as possible.
• A test is successful (a good test) if it found an error; bad tests find few/no errors.

The seven ISTQB testing principles

• 1. Testing shows the presence of errors — it cannot prove their absence (no proof of error-freeness).
• 2. Exhaustive testing is impossible — tests are samples; set effort by risk/priority (higher risk → more tests).
• 3. Start testing early — early detection of errors.
• 4. Defect clustering — errors concentrate in few parts; where one is found, more hide nearby.
• 5. Repetition is ineffective — repeating the same test cases yields no new findings (pesticide paradox).
• 6. Testing is context-dependent — adapt to the test object and its environment.
• 7. Absence-of-errors fallacy — removing all errors does not mean the system meets users' needs.

Test levels, test types, and testing roles

Systems are built by "divide and conquer": split the task into manageable parts with verifiable hand-over points, each tested independently. The test levels (bottom-up) are Component (Unit) test, Integration test, System test, Acceptance test; each level contains several test types and each type needs its own methods and tests one or more quality criteria.

• Test types include: functional tests; performance tests; stress test (behavior under overload); mass/volume test (large data); reviews; and more.
• Component test (Unit test): checks separately testable components (modules, objects, classes) in isolation using test doubles and test drivers; found errors are clearly attributable; test cases derived from development documents; needs debugging tools or a component-test environment (e.g. JUnit).
• System test: focuses on the whole system's specified behavior in an environment equivalent to production (even minimal config differences can cause production problems despite passing tests); broad coverage of functional and non-functional requirements; usually the last full check before delivery. Smoke test: minimal, partly random tests ensuring no totally unusable version is released (does it start? does it crash?).
• Test manager: interface between test team and project management; responsible for a working test process; plans testing with project management and assigns work packages; controls progress and writes the test report.
• Test engineer (Testingenieur): turns the test manager's strategy (which/how many tests, what focus) into test cases.
• Test developer: realizes the test cases designed by the test engineer.
• Tester: actually runs the tests using the test cases and the manager's specifications; test logs feed corrections, reporting and progress control.
• Test strategy: an approach for efficient, economical tests, set in the test plan — defines test types per quality attribute, their order, and test intensity (focus on areas with high error probability).

Acceptance test

The acceptance test proves the contractor delivered what was agreed; finding errors is not its goal. It is usually done by the customer/ users and answers: does the customer think their requirements are met? It need not be the last level (a system-integration test may follow). Requirements-elicitation mistakes are usually found here, but since the software is far advanced, fixing them is expensive — mitigate with mini-acceptance tests of partially finished components.

• Aspects: user acceptance test (usability by users); operational acceptance test (admin: backup/restore, recoverability, user management, maintenance, periodic security checks); regulatory & contractual acceptance test (contractual criteria set at contract close; regulatory checks legal/standard compliance).
• Alpha test: at the manufacturer's site; Beta test / field test: at the customer's site (beta versions given to a broad customer base to find errors before the final release). Both are done by customers, not developers.

Regression testing

In a regression test the previous version of the system defines the expected (presumably correct) behavior, rather than a classic specification. Used when the system changes (component swap, code change) to check whether previously-given functionality still holds — "does my software behave identically before and after changes?". The more complex the system, the greater the danger that a small intended change causes unforeseen side effects.

• Especially suited to automation: regression cases need no change as long as only the implementation changes, not the interfaces; cases can even be auto-generated (random data, record returns).
• Back-to-back test: using regression as a comparison tool (version n vs n-1). Re-test: re-running all tests after a fix to confirm the error is really gone.

Control-flow coverage measures (C0–C4)

Structural (white-box) methods follow the control-flow graph; full 100% coverage is often impossible or unnecessary (e.g. for stable libraries) — identify the elements critical to functionality.

• C0 — Statement coverage: every statement (node) is executed at least once. Weak (too easy: one execution counts as 100% tested) and misses interactions, but fast to compute and good for finding completely untested code.
• C1 — Branch/edge coverage: every branch (edge) of the control flow is traversed at least once — all true/false outcomes (an if without else still has 2 edges), all case/switch variants, all exceptions/returns. C1 guarantees C0 but not vice versa; weakness: one loop pass counts as a full loop test, and complex conditions are evaluated only as a whole.
• C2 — Simple condition coverage: each atomic sub-condition must take both truth values (considered individually). 100% C2 does not guarantee C0/C1, and short-circuit evaluation (||/&&) can leave a sub-condition's evaluation unchecked despite 100% C2 — so a fault in it may be missed.
• C3 — Multiple condition coverage (branch condition combination): also checks all composite sub-decisions (truth tables); extremely costly — 2^n test cases for n sub-decisions. Minimal multiple-condition coverage (MC/DC): only combinations where changing one atomic condition changes the overall condition.
• C4 — Path coverage: executes all distinct paths through the test object (a path = a unique sequence from entry to exit), considering dependencies like loops. The number of paths explodes (10 ifs → 1024 cases; loops → unbounded), so decide per system part where it is really needed.

Coverage traps — coverage is necessary but not sufficient

The primary use of coverage is to answer "have I tested enough?", but high coverage does not mean good testing.

• The 'tester walks into a bar' joke: a tester orders 2 / 0 / 99999999 / a lizard / -1 / 'qwertyuiop' beers — covering many edge inputs — but a real customer asks where the toilet is and the bar burns down: tests can achieve high coverage yet miss whole classes of real-world inputs/scenarios.
• Triangle-test lesson: it is hard to enumerate all relevant cases (valid: equilateral, isosceles, right-angled, scalene; invalid: triangle inequality violated, degenerate, negative side, wrong parameter count, zeros) even though counting coverage afterwards is easy.
• Combine coverage with a 'does it find errors?' view: prefer measuring detected (real or mutated) defects, not coverage alone, to judge sufficiency. Note: this course's coverage definitions differ from Wikipedia — use the LV definitions.

Functional and informal test-design methods

Equivalence-class analysis partitions input/output values by system behavior into classes whose values are assumed equivalent: if one value triggers (or doesn't trigger) an error, all values in its class do too. Each class is tested at least once with a representative, reducing test effort.

• Distinguish valid and invalid equivalence classes — invalid classes are not arbitrary corrupted data but values outside the expected range. Test cases must cover both valid and invalid inputs.
• Example (calendar months 1–12): valid class 1–12; two invalid classes <1 and >12. Example (age >18 and <65): three classes A1 ≤18 (invalid), A2 >18 and <65 (valid), A3 ≥65 (invalid). Example (age >40 AND BMI >25): two classes each → four combinations.
• The most common error source is the boundaries of the classes — leading to boundary-value analysis.
• State-based testing: based on state machines (UML state diagrams); error cases must be specified separately; completeness criteria are state coverage, transition coverage, and event coverage. Used for technical applications, GUIs, and state-machine-defined systems.
• Classification-tree method: partition inputs/states into classifications, split into disjoint classes, combine classes into test cases (each class once per classification). Combinations: minimal (each class once), maximal (every class with every other), pairwise, tripleweise.
• Informal methods: wholly/partly skip systematic case derivation, rely on tester intuition/experience — not exactly repeatable or deterministic, but often effective and cheap for spotting fault-prone areas (then tested formally). Exploratory testing: simultaneous learning and testing. Ad-hoc test: run once, positive results not logged. Not suited for repeatable/automated/documented or regression tests.

Boundary-value analysis

Boundary-value analysis is a special case of equivalence-class analysis: misbehavior often occurs at system boundaries (the upper/ lower ends of a class), e.g. an off-by-one error writing ≥ instead of >. Identify the class boundaries and choose test data from just around them — one value on each side of a boundary, and the boundary value itself (which must belong to one class).

• Example (age >18 and <65): sensible representatives are 18 (invalid, A1), 19 (valid, A2), 64 (valid, A2), 65 (invalid, A3).
• Theory: at most two boundary values per class (one per upper/lower edge); practice: possibly more per boundary (e.g. also 17 and 66), trading off finding errors vs test effort — needs experience with the system.

Test data, data-driven tests, and positive vs. negative tests

• Test data configure the target system, test doubles and test drivers, and drive the driver during execution (a data-driven test).
• Data-driven test: separates test cases from test data — cases are defined with placeholders and bound to concrete data only at execution. Benefits: cases and data reused independently; better maintainability/readability; m:n use (one test, many data sets); cases definable before concrete data exist; easier, more maintainable automation.
• Positive test (most common): checks the system behaves correctly for valid input (e.g. div(2,2) returns 1). Negative test: explicitly checks that problems are (a) detected and (b) handled as expected, e.g. div(2,0) throws the expected ArithmeticException.

Readable tests — naming, Gherkin/Cucumber

Readable test names follow a scheme like WhenWeAreInStateX_AndSomethingHappens_ThenTodoIsDone(), so a test result immediately shows what works or not. Cucumber (test framework) and Gherkin (Given/When/Then syntax) refine this idea.

• A Gherkin Scenario Outline declares Given/When/Then steps plus an Examples table (e.g. day → answer: Friday→TGIF, Sunday→Nope); Cucumber step definitions (@Given/@When/@Then) bind each step to code and assert the expected vs actual answer.
• This is a data-driven test: one scenario runs for every Examples row. Advantage over classic unit tests: the Given/When/Then specification is readable by non-technical stakeholders (business/customer), bridging requirements and tests, while reusing one scenario across many data rows.

Test-case documentation and best practices

• Documentation enables a systematic, transparent, traceable test process and helps debugging; it spans all levels from component to system tests.
• A test case contains: a running ID; type (normal NF, special SF, error FF); preconditions; input values; description (which component, which property); expected result; actual result; decision (do expected and actual match?).
• Best practices (worksheet 3.10): keep tests small and well-organized; ensure reproducibility and side-effect-freeness; abstraction & isolation; avoid Thread.sleep (flaky, slow — it is 'evil'); use frameworks and keep tests green; handle unexpected exceptions explicitly. Unprepared/undocumented tests are largely useless because they are not repeatable or traceable.

Traditional testing vs. TDD

Traditional testing finds errors in an already existing solution, testing late (toward/after the end of development) — drawbacks: errors are found late and are harder/costlier to fix, sometimes skipped under time pressure ('banana software' that ripens at the customer), and it needs complete requirements/specifications.

In TDD, test cases are specified before or in parallel with implementation — the system is effectively designed via tests; it is the best practice in agile development. Benefits: immediate feedback, early detection of problems/side effects, and a plausibility check of requirements/specifications. In practice: higher customer satisfaction, fewer customer-reported errors, and a shorter test phase that offsets the initial extra effort. (Worksheet account example: implement Account with getAmount/deposit/withdraw/transfer test-first, a minimal set of tests per method.)

Fuzzing

Fuzzing automatically generates test inputs to find problems with low effort. A fuzzer feeds (mutated/generated) inputs to the program and watches for crashes/abnormal behavior; good seed selection matters (cf. "Optimizing Seed Selection for Fuzzing", Rebert et al.).

• Easily finds crash-type defects (e.g. inputs that cause exceptions/crashes) regardless of refactoring; cheap and good when a project lacks a sufficient test base (e.g. legacy projects).
• For regression support, the recorded behavior/outputs must serve as the expected baseline (record returns of the old version, compare after changes).
• Limits: mainly finds crashes/robustness issues, not functional-correctness bugs without an oracle; cannot replace targeted manual unit tests.

Asserts — pre/post-conditions, invariants, weak vs. strong

Beyond "big" unit tests, assert statements add small in-code mini- tests (e.g. assert(var != null) throws automatically if violated) and are used broadly (Microsoft Office has over a million). They check pre-conditions, post-conditions and invariants (the same Design by Contract concepts as in the Implementation block).

• Pre-condition: required state at/before a call; Post-condition: state at method end; Invariant: stable object state between calls and after construction (Stack example: pre top>0 before pop, post top decreased, invariant 0 <= top <= capacity).
• Weak vs strong negotiation: as the method implementer you prefer strong pre-conditions (caller must guarantee more, less work for you) and weak post-conditions (you promise less); the caller prefers the opposite (weak pre, strong post).
• They need not be asserts (also possible via comments or frameworks). Asserts vs unit tests: asserts are always-on internal checks across many runs and document contracts, but are not standalone test cases; unit tests are external, targeted, repeatable cases — they complement each other.

Automated and non-functional testing

Test automation is the systematic, repeatable, traceable, tool-supported running of predefined tests. About 30–50% of development costs go to testing, so it is a professional, ongoing process integrated into development.

Non-functional tests are specialized for the most important non-functional properties (performance, usability, security); they need more expert and domain knowledge than functional tests and are usually run by a separate, specialized team.

• Pros: once created, tests run arbitrarily often with little extra effort; running effort is far lower than manual. Cons: building automation concepts is expensive (preparation ~2× the manual effort) and needs a suitable framework (e.g. JUnit). Applied to regression tests and CI strategies.
• Automated component tests use unit-test frameworks that define and continuously run tests in the IDE (dynamic result evaluation), provide verification methods, and are often coupled with code-coverage frameworks (instrumentation) — frequently targeting ~80% coverage.
• Automated GUI tests simulate the user through the UI (a black-box approach where both the Point of Control and Point of Observation are the UI; higher complexity). Capture/Replay: record user actions into a script — fast to create but poorly maintainable and only usable once the (near-final) GUI exists. Scripting: write tests from scratch (UI elements by ID) — slower to create but more maintainable. Practice: combine both (e.g. manual rework of recordings).
• Performance test: a risk-minimizing test of performance/scalability against SLAs (measurable required performance); run regularly to localize which change degraded performance. Key metrics: latency (response time), throughput (data/work per time), transaction rate (a special throughput). Load test: a special case checking correct function under higher load. Performance tests record behavior (monitoring) instead of functional assertions.
• Usability testing: improves product usability; representative users perform realistic tasks while usability experts observe and record, then analyse and propose improvements; phases: preparation/planning, execution, evaluation, communication of results (written report, e.g. NIST Common Industry Format).
• Security test: checks security via methods like penetration testing (attacking a running system with various techniques to find weaknesses); identify unspecified functionality as it can be a vulnerability — detailed in the Security chapter.

Organizational quality assurance

Organizational QA provides infrastructure to avoid software errors or at least lower the error rate.

• Knowledge management: a development of organizational learning, improving the organization at all levels through deliberate handling of the resource knowledge.
• Configuration management: manages all objects (specifications, documentation, source) across the whole lifecycle; must control version lines, change states and releases of the 'configuration items'.
• Templates: secure the format, structure and content of documents (test plans, test-case descriptions, test/review reports).
• Checklists: guidelines and hypotheses about suspected weaknesses, phrased as yes/no questions.

Quality standards and formal verification

Quality management is the set of coordinated activities to direct an organization regarding quality; standards raise the quality level and continuity, improve mutual understanding/coordination, and increase customer confidence. A distinction is made between certification and assessment standards (certification can be required for some state projects). Testing standards include ISO/IEC/IEEE 29119 (concepts, processes, documentation, techniques, keyword-driven testing); test certifications (ISTQB, ISAB) are increasingly popular.

Source code can be translated into formulas so that SAT (Boolean satisfiability / propositional logic) or SMT (satisfiability modulo theories / predicate logic) can check whether the program shows the expected behavior.

• ISO 9001 / ISO 9000-3: international quality-assurance norms; process-centred, oriented to customer needs, certification-focused.
• CMM / CMMI (Capability Maturity Model[/Integration]): maturity models rating the quality of an organization's development processes, developed by the SEI for US-DoD supplier comparison; CMMI is the unified, modular successor.
• SPICE (ISO/IEC 15504): structured very similarly to CMMI, a joint ISO/IEC initiative and competitor to the US-dominated CMMI.
• Approach: translate the first n lines into a formula N, translate the required properties into a formula A, and check whether some variable assignment makes N → A true; if ¬(N → A) is satisfiable, an error exists and the assignment is a clue. Solvers evaluate SAT/SMT formulas automatically.
• Advantage over tests: formal verification can give a final verdict on whether a system is error-free, so it is applied to limited critical areas (e.g. parts of rocket-control software). Disadvantage: very costly, so not broadly usable.
• The Z notation is a formal ISO-standard language (since 2002) to define systems via declarations (above the line) and conditions (below), e.g. modelling a debit from an account.