1. LandOffer
  2. Blog
  3. All articles
  4. A Test Passes Locally but Fails in CI: Reproduce the Environment Difference

Engineering Practice

A Test Passes Locally but Fails in CI: Reproduce the Environment Difference

An actual four-process Python fixture isolates a working-directory mismatch, preserves a missing-file failure and checks a module-anchored repair without claiming a real CI run.

9 min readLandOffer team

When a test passes locally but fails in CI, preserve the failure and compare the effective environments before changing the assertion. Confirm the artifact, runtime, dependency installation, command, working directory, configuration and external resources. Reproduce one material difference in an isolated environment, inspect whether the same failure appears, then test the repair under both conditions. A passing local retry does not explain a different CI environment.

This LandOffer.ai guide includes an environment-comparison record and an actual four-process Python demonstration. The original synthetic fixture uses temporary directories and harmless local configuration. It did not run GitHub Actions, install dependencies, access credentials or test another operating system. Sources were checked October 27, 2026; the observed interpreter is Python 3.12.14. “Worker directory” below describes a local test condition, not an observed hosted runner.

An archivist traces two entrances to the same records room, comparing where a search begins with where the records are stored.

Preserve what failed and what actually ran

Retain the failing test identifier, exact command, exit status and relevant full output. Identify the source revision and any generated artifacts, fixtures or configuration used by that command. A branch name or test label is not enough if local uncommitted files, ignored resources or a different built package participated in the passing run. Retain the original failure alongside the later successful result so a reviewer can compare both.

Compare the command after its shell, wrapper and working-directory settings have been applied. A shell, wrapper script or task runner can choose a different interpreter, working directory or group of tests. A local editor can also supply settings that an ordinary terminal does not. Check what actually executed so the investigation can target a supported difference instead of changing an unrelated test setting.

Record versions without collecting unrelated private information. Runtime and package-manager versions, an approved dependency inventory and relevant nonsecret configuration can help. Do not dump the entire environment or credential-bearing URLs into logs. For a required secret, record whether the authorized mechanism supplied it and which permission scope is expected; the value itself is not diagnostic evidence to publish.

A deterministic mismatch is different from an intermittent test. If one directory consistently produces a missing-file error and another consistently succeeds, investigate that directory dependency. Repeating the successful condition cannot disprove the failure in the other condition. Keep state-leak, timing and external-service hypotheses available only where the evidence supports them.

Compare dependency and configuration inputs before rebuilding

Check the lockfile, package manifest, package-manager version and installation flags as a set. A developer's existing dependency directory can contain packages that a fresh installation would not provide. Conversely, a CI installation may intentionally omit development dependencies or use a different workspace selection. Those are concrete differences to verify rather than reasons to update every dependency.

The npm v11 npm ci documentation describes a clean installation using an existing lockfile. It fails on a manifest/lock mismatch instead of updating the lock and requires matching tree-affecting flags. This is version-specific documentation, not an npm operation executed for this Python example. A frozen installation also does not make operating-system libraries, configuration or external services identical.

For a real investigation, reproduce installation using the project's approved procedure in a disposable workspace. Preserve its result before changing a lockfile or clearing a shared cache. A failed install and a failed assertion occur at different stages; do not describe the test as having exercised application behavior if dependencies never installed or collection never completed.

Compare time zone, locale, encoding, filesystem case behavior, permissions and required environment variables when they affect the failure. Name which difference is observed and which is still only a hypothesis. A Linux runner and a macOS workstation can differ in relevant ways, but the platform names alone do not identify the cause. This article's experiment keeps the host and interpreter fixed so it can isolate a smaller difference.

Completed asset: an effective-environment comparison

Our synthetic project contains config_reader.py, settings.json and test_mode.py. The JSON contains only {"mode":"fixture"}. The test imports load_mode() and expects the literal value fixture. There are no third-party dependencies. An independent empty directory stands in for a job that starts somewhere other than the project folder.

The original function opens Path('settings.json'), so its relative lookup depends on the process's current directory. Python's pathlib documentation describes the current directory and resolution behavior. The actual experiment below checks this particular file lookup rather than relying on the documentation to claim an unobserved failure.

Effective condition Project-directory run Empty-worker-directory run
Interpreter Same observed Python 3.12.14 executable Same executable
Application and test contents Same original source and test hashes Same hashes
Configuration content and location Synthetic JSON beside the module Same file beside the same module; no second config in the empty directory
Module import path Explicit path to the synthetic project Same explicit path
Process environment Minimal explicit nonsecret values Same values
Current working directory Synthetic project directory Separate empty directory
Observed original result Exit 0; expected value returned Exit 1; file lookup raises FileNotFoundError

This is a completed comparison of observed local conditions, not a fabricated CI log. Every command starts a fresh subprocess. The explicit import path allows the test and module to load from either starting directory; therefore the second failure is not an inability to import the test. The retained error log names the selected test, the call to load_mode() and the missing settings.json lookup; it reaches the intended resource-loading path.

GitHub's workflow-defaults documentation explains that run steps can have shell and working-directory defaults, with more specific settings overriding broader defaults. That is a reason to inspect effective settings in an actual Actions investigation. It does not mean this fixture discovered a GitHub default or reproduced a hosted VM.

The two runs do not compare package versions, time zones or filesystem case sensitivity; those inputs were not varied. No actual dependency installation was needed because the fixture uses only the standard library. Fresh temporary files and processes give this experiment a clean local starting point, while leaving the OS, interpreter installation and kernel shared with the host.

Completed asset: reproduce the difference and test a bounded repair

The original module is intentionally small:

import json
from pathlib import Path

def load_mode():
    return json.loads(Path('settings.json').read_text())['mode']

The test calls that function and asserts that the result equals fixture. In both original runs, the driver invokes the same named unittest method with the same interpreter and explicit module search path. From the project directory it passes. From the empty directory it reaches the function and fails with a missing-file exception identifying settings.json.

That evidence supports a specific diagnosis: the fixture's resource lookup depends on the starting directory even though its stated configuration belongs beside the module. It does not establish that every missing-file error has this cause. A missing packaged resource, wrong filename, permissions failure or omitted checkout file would need different evidence and a different repair.

The example's revised function uses its own module location:

def load_mode():
    config = Path(__file__).resolve().parent / 'settings.json'
    return json.loads(config.read_text())['mode']

The saved implementation uses the equivalent expression inline. The important change is the lookup anchor, not the temporary variable's name. The same test file is retained unchanged. The driver runs the revised module from the project directory and from the empty directory, each in a fresh process. Both return exit 0 and the expected value.

Actual local run Source version Outcome Interpretation
1: project directory Original relative lookup Pass The original succeeds when its implicit starting-directory requirement happens to hold
2: empty directory Same original source Missing-file error Changing only the starting directory exposes the hidden requirement
3: project directory Module-anchored repair Pass The previously successful condition remains supported in the fixture
4: empty directory Same repaired source Pass The tested lookup no longer depends on that starting directory

The retained logs, source hashes, test hash and configuration hash bind those results to the exact fixture. The four exits, in the table’s order, are 0, 1, 0, 0; the error belongs to the original source launched from the empty directory. They are not four independent incidents or evidence about a real service's failure rate. The first failed command remains saved rather than being replaced by the two green results.

A file path starts from an empty working folder in one case and from the module's own folder in another, with the configuration kept beside the module.

Choose the repair from the configuration contract

Anchoring to the module is appropriate here because the example explicitly defines the JSON as a regular file distributed beside it. It is not a universal application configuration pattern. A packaged resource may need the language's resource-loading API; an operator-supplied file may need an explicit validated path; a command-line tool may deliberately interpret a relative path from its user's directory.

If the intended contract requires a project-root working directory, an effective workflow correction might be the appropriate change instead. Preserve that requirement in the supported invocation and test it. Do not silently add several fallback search paths that might load the wrong file and make a test pass while changing configuration precedence. Finding some file is different from finding the intended file.

Keep the failure mode meaningful. Replacing a missing required configuration with an arbitrary default can mask a deployment omission. In this fixture, we changed the anchor while preserving the required file and expected value; we did not add a catch-all exception or change the assertion. Missing-file behavior outside the two exercised conditions remains a separate contract question.

For the actual handoff, state which explanation is supported and what remains unverified. Here the completed record is: same original source, test, config and interpreter; different current directory; missing-file failure reproduced; module-location lookup passes both starting conditions. A real workflow change and its hosted rerun have not occurred. The local result is ready for review, not evidence that an actual CI incident has been resolved.

Verify the real boundary before closing the issue

In a real project, carry the supported repair into the approved workflow and verify it against the original failing command and environment. Check the expected package or resource actually reached the runner, the intended interpreter was selected and the same test was exercised. Inspect the full relevant output, including errors before test execution. A green job that skipped the affected tests does not close the original problem.

If local reproduction still passes under the reconstructed conditions, keep the mismatch open. Compare the remaining observed differences and improve the reproduction rather than weakening the test to match an unexplained CI result. Record unavailable runner evidence or inaccessible dependencies precisely. Keep the observed failure and unresolved environment fields visible instead of attributing the mismatch to an unverified cache or operating-system cause.

Use one failing test to create an effective-environment comparison, an isolated reproduction and a repair record with its remaining gaps. LandOffer.ai publishes this engineering guide; it does not inspect your CI account or change workflow settings. Before closing the actual issue, give the decision owner the original failing path, the verified result under the intended environment and the remaining coverage limits.

Sources and Further Reading