1. LandOffer
  2. Blog
  3. All articles
  4. Code Coverage Went Up: Check Which Behaviors Are Still Untested

Engineering Practice

Code Coverage Went Up: Check Which Behaviors Are Still Untested

An executed synthetic seat-eligibility predicate reaches complete statement and branch coverage with weak boolean-type checks. Six contract-derived assertions detect three deliberately changed implementations that those checks accept. Full coverage reports, behavior tables, and failure receipts distinguish execution from useful protection.

9 min readLandOffer team

When code coverage rises, inspect the new tests before treating the increase as stronger protection. Separate the statements they execute, the branch destinations they visit, and the behavior their assertions would reject. Then choose a plausible wrong implementation and check whether the suite notices it. A percentage can locate an execution gap; it cannot establish that a test’s expected result is correct.

This guide from LandOffer.ai builds a complete local example in which additional tests reach every measured statement and branch while missing three deliberately introduced mistakes. Explicit expected-value assertions detect these three mistakes while the measured coverage figure stays unchanged. The example is synthetic, the Python runs are real, and none of the results describe an employer’s application or release.

A reviewer separates executed code paths from an expected behavior record, with a boundary case still requiring an assertion.

Read the coverage change as a question about evidence

Read the report’s scope and denominator first. Which files, generated code, test types, and exclusions were included? Did the increase come from new tests, a broader integration run, removed source, or changed reporting settings? Comparing percentages across different scopes can hide what actually changed. Retain the source revision, test command, runtime, and coverage configuration beside the two reports.

Statement coverage records execution of executable statements. Branch coverage adds information about alternative control-flow destinations. Neither measure, by itself, tells you whether a test compares the returned value with the right expected value. A test can invoke a failure path and ignore the resulting status, message, or state.

Coverage.py’s branch documentation explains that it tracks transitions between source lines and compares observed destinations with possible ones. Its combined coverage figure includes statement and branch opportunities. Read the separate statement and branch counts before describing a result simply as “100% coverage.”

The testing overview in Software Engineering at Google similarly distinguishes executing code from verifying useful behavior. Use that distinction in review: “Which incorrect result would this assertion catch?” is more revealing than “Which extra call makes this line green?”

Coverage remains useful. An unvisited rejection path can identify a missing case quickly. The mistake is letting a complete execution map replace the work of defining expected behavior. Inspect a small, important module first, especially one whose wrong output could quietly change a user’s decision.

Worked example: complete execution with weak assertions

The fictional SeatWindow tool helps a workshop coordinator decide whether a requested number of seats fits within current availability. Its pure function returns a boolean. For this teaching contract, inputs are integers, availability is nonnegative, a paused workshop rejects requests, and a request must be positive and no larger than availability. Equality is allowed: three requested seats can fit into three available seats.

That contract was written before selecting tests. It does not describe a real booking system. The function neither reserves inventory nor checks identity, payments, concurrent requests, or database state. Those boundaries require different evidence if a product uses such a predicate.

The complete implementation is short:

def can_reserve(available, requested, paused):
    if paused:
        return False
    if requested <= 0:
        return False
    return requested <= available

The initial test checks one ordinary request: two seats out of three available, with the workshop open, returns True. It makes a useful assertion about that case. It does not visit the paused or nonpositive-request returns, so the execution report correctly exposes missing paths.

Now add three calls: a paused request, a zero-seat request, and the original ordinary request. The expanded test asserts only that each result is a boolean:

for args in [(3, 1, True), (3, 0, False), (3, 2, False)]:
    assert isinstance(can_reserve(*args), bool)

This suite executes both outcomes of each if and reaches each return. It still accepts the wrong boolean. Returning True for a paused workshop satisfies the type assertion just as easily as the correct False.

The isolated run used Python 3.12.14 and Coverage.py 7.17.0, installed into a disposable directory rather than this project’s environment. The measured source was only capacity.py; the test file was outside that source filter. Commands, source snapshots, JSON reports, and outputs are retained with the draft. The temporary installation and coverage database were removed afterward.

Suite What it actually asserts Statement result Branch result What remains missing
Initial Ordinary two-of-three request returns True 4 of 6 statements executed 2 of 4 branches executed Paused, invalid, equality, and over-capacity behavior
Expanded Three calls return a boolean 6 of 6 statements executed 4 of 4 branches executed Correct values, equality boundary, and excess request rejection
Stronger Six explicit inputs match six contract-derived booleans 6 of 6 statements executed 4 of 4 branches executed Product integration, concurrency, and inputs outside this fixture contract

The initial combined report is 60%: four executed statements plus two visited branch destinations out of ten total opportunities. Its statement-only figure is about 67%, and its branch figure is 50%. The expanded report reaches 100% in all three measures. Those are actual Coverage.py results for this file and configuration, not interchangeable percentages from different dashboards.

The expanded suite really exercises more code, but its type checks still accept incorrect decisions. Its interpretation was too broad if a reviewer called it proof that seat eligibility works. The first completed review asset is this table plus the retained report pair: it explains both the improvement and its remaining blind spot.

Write the missing behavior record before adding more calls

Separate a missing path from a weak assertion on an already visited path. The paused return is already executed, but its meaning is unverified. Equality uses the same final return as ordinary availability, yet deserves its own input because changing <= to < alters the contract without adding a new source line.

Here is the filled behavior record used for the stronger suite. Each expected value follows the stated contract, rather than being copied from a run of the implementation.

Case Available, requested, paused Expected Why this case belongs
Paused 3, 1, True False A paused workshop rejects even an otherwise fitting request
Zero 3, 0, False False A zero-size request is not a reservation request
Negative 3, -1, False False A negative quantity must not be accepted as spare capacity
Below capacity 3, 2, False True An ordinary positive request fits
Exactly capacity 3, 3, False True The agreed comparison includes equality
Over capacity 3, 4, False False A positive request can still exceed availability

The stronger test compares each result with its explicit expected boolean and includes the case name in a failure. All six cases pass on the baseline implementation. Coverage remains complete for the same measured file; the useful change is stronger behavioral checking, not a higher percentage.

For a real change, ask the relevant product or domain owner to settle ambiguous expectations. Should a request of zero be invalid, ignored, or represented separately? Should an unavailable item return a rejection reason? A test that freezes an accidental implementation choice can make future correction harder. Record unresolved policy questions instead of inventing an answer solely to complete a test table.

Use observable contracts rather than private implementation details wherever practical. In this fixture the boolean is the contract. In a service, the relevant observation might include an HTTP response and the absence of a stored reservation. A mocked success response alone would not prove that the database stayed unchanged after a rejected request.

Check whether plausible mistakes survive

Mutation testing asks whether deliberately changed code makes tests fail. Stryker’s state documentation distinguishes mutants detected by failing tests from mutants that survive, and also identifies uncovered, invalid, and timed-out runs. Read those categories before treating a tool’s score as a simple quality grade.

For this article, three explicit one-change variants were run manually in isolated Python processes. Stryker was not installed or executed, and the three selected variants are not an exhaustive mutation campaign. Each changed implementation was tested against both the expanded and stronger suites.

Deliberate change Expanded boolean-type suite Stronger expected-value suite Relevant failure
Replace final <= with < Passes; mistake survives Fails Exactly-capacity request should be accepted
Return True when paused Passes; mistake survives Fails Paused request should be rejected
Replace nonpositive check with negative-only check Passes; mistake survives Fails Zero request should be rejected

Each retained failure output identifies the expected-value assertion and the wrong boolean, rather than a broken runner or invalid source. A syntax error, missing import, or broken test runner would not establish that the behavior assertion detected the mistake. The retained records include successful baseline runs and nonzero exits for these specific changed implementations.

This is the second completed review asset: a behavior table connected to actual counterexample results. It makes a narrower, stronger statement than “more tests were added.” The revised suite rejects three demonstrated mistakes that the previous suite accepted, while preserving the same coverage scope.

A surviving mutant requires interpretation. It may reveal a missing input, an assertion that ignores output, or a change that is equivalent under the valid input contract. Investigate which explanation applies. Do not add a brittle assertion merely to remove a survivor, or quietly exclude difficult cases to improve a score.

Three deliberately changed implementations survive a type-only suite but fail explicit behavior assertions; this is a limited manual check rather than a complete mutation score.

Finish with a bounded testing decision

The fictional review note is complete: “Keep the six explicit behavior cases. The initial-to-expanded coverage increase reached previously unvisited rejection paths, but type assertions accepted three demonstrated mistakes. The stronger suite detects those mistakes. This supports the stated integer-input predicate only; it does not establish a safe reservation workflow.” This is an authored example, not a real approval or team message.

The remaining work has owners in the example plan. The application developer must check how the caller represents rejection; the service owner must assess concurrent availability changes before using the result to commit inventory. Their integration checks are pending, not implied by the local run. Neither the article nor the test result authorizes a product release.

The example deliberately leaves additional questions visible. Zero availability, unusually large integers, booleans passed as quantities, and non-integer inputs need a contract decision before they become acceptance claims. The stated input domain does not make an unsafe public API safe. If the caller can send other types, decide where validation belongs and test that boundary separately.

Prioritize the failure that could quietly approve an action the user should not take. Here, allowing paused or zero-size requests is more important than proving a private helper was invoked a particular number of times. In another module, preserving stored data or avoiding a duplicate side effect could dominate. Make that choice explicit instead of assuming every uncovered line has equal risk.

When reporting progress, keep execution metrics and behavior evidence next to each other. Include the changed scope, new assertions, important boundaries, a representative failure the suite now catches, and unresolved risks. If coverage increased because files were excluded or denominator settings changed, say that directly rather than attributing the increase to stronger tests.

Choose the next test by the consequence of a wrong result and the uncertainty in the contract. The best next case may exercise an already covered line with a previously untested boundary. Choose that case with the code’s owners; this fixture establishes no universal coverage threshold or forecast of prevented defects.

For your next coverage review, open one newly covered function and ask which wrong output would still pass. Write down the expected behavior, demonstrate one meaningful counterexample, and keep the result with the report. More practical engineering guides are available from LandOffer.ai.

Sources and Further Reading