Skip to content

Enable godog strict mode to fail on undefined steps#3111

Merged
st3penta merged 2 commits intoconforma:mainfrom
st3penta:strict-acceptance
Feb 13, 2026
Merged

Enable godog strict mode to fail on undefined steps#3111
st3penta merged 2 commits intoconforma:mainfrom
st3penta:strict-acceptance

Conversation

@st3penta
Copy link
Contributor

Add Strict: true to godog.Options to ensure undefined steps cause
test failures instead of being silently ignored. This prevents false
positives where tests appear to pass despite missing step definitions.

This commit also reduces logging verbosity in acceptance tests.

st3penta and others added 2 commits February 12, 2026 12:14
Add Strict: true to godog.Options to ensure undefined steps cause
test failures instead of being silently ignored. This prevents false
positives where tests appear to pass despite missing step definitions.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@qodo-code-review
Copy link
Contributor

Review Summary by Qodo

Enable godog strict mode and reduce test logging verbosity

✨ Enhancement 🧪 Tests

Grey Divider

Walkthroughs

Description
• Enable strict mode in godog to fail on undefined steps
• Prevent false positives from silently ignored step definitions
• Reduce logging verbosity by removing scenario start messages
• Simplify output formatting with single newlines
Diagram
flowchart LR
  A["godog.Options"] -->|Add Strict: true| B["Fail on undefined steps"]
  C["Logging output"] -->|Remove start messages| D["Reduced verbosity"]
  E["Output formatting"] -->|Single newlines| F["Cleaner output"]
Loading

Grey Divider

File Changes

1. acceptance/acceptance_test.go 🧪 Tests +3/-8

Enable strict mode and reduce logging verbosity

• Added Strict: true to godog.Options in TestFeatures function to enforce undefined step
 failures
• Removed scenario start logging that wrote to /dev/tty to reduce verbosity
• Simplified output formatting by removing extra newlines from scenario end messages
• Kept scenario failure/success logging for essential test feedback

acceptance/acceptance_test.go


Grey Divider

Qodo Logo

@qodo-code-review
Copy link
Contributor

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (1) 📎 Requirement gaps (0)

Grey Divider


Remediation recommended

1. Unstructured /dev/tty logging 📘 Rule violation ✧ Quality
Description
The acceptance test writes plain-text status lines directly to /dev/tty, which is unstructured and
harder to parse/monitor consistently. This can violate the requirement that logs be structured for
auditing/debugging purposes.
Code

acceptance/acceptance_test.go[R142-144]

+				fmt.Fprintf(tty, "✗ FAILED: %s (%s)\n", scenario.Name, scenario.Uri)
			} else {
-				fmt.Fprintf(tty, "✓ PASSED: %s (%s)\n\n", scenario.Name, scenario.Uri)
+				fmt.Fprintf(tty, "✓ PASSED: %s (%s)\n", scenario.Name, scenario.Uri)
Evidence
PR Compliance ID 5 requires structured logs; the changed lines emit free-form text via fmt.Fprintf
to /dev/tty, which is not structured output.

Rule 5: Generic: Secure Logging Practices
acceptance/acceptance_test.go[142-144]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Acceptance tests write unstructured plain-text status messages to `/dev/tty`, which conflicts with the requirement that logs be structured for easier auditing and monitoring.

## Issue Context
The PR modified the scenario result output (PASS/FAIL) but it remains free-form `fmt.Fprintf` output.

## Fix Focus Areas
- acceptance/acceptance_test.go[142-144]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Lost scenario start logs 🐞 Bug ✧ Quality
Description
The PR removes the per-scenario STARTING log and only logs PASSED/FAILED at scenario completion. If
a scenario hangs or the process crashes mid-scenario (especially with parallel execution), there’s
no tty output indicating which scenario was running, slowing triage.
Code

acceptance/acceptance_test.go[R132-136]

		logger, ctx := log.LoggerFor(ctx)
		logger.Name(sc.Name)

-		// Log scenario start - write to /dev/tty to bypass all output capture
-		if tty, err := os.OpenFile("/dev/tty", os.O_WRONLY, 0); err == nil {
-			fmt.Fprintf(tty, "\n▶ STARTING: %s (%s)\n", sc.Name, sc.Uri)
-			tty.Close()
-		}
-
		return context.WithValue(ctx, testenv.Scenario, sc), nil
	})
Evidence
The Before hook no longer writes a STARTING marker; the only tty logging happens in the After hook,
which will not run if the scenario hangs or the process terminates mid-run.

acceptance/acceptance_test.go[131-136]
acceptance/acceptance_test.go[138-147]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Scenario STARTING logs were removed, leaving only end-of-scenario PASSED/FAILED logs. When running with parallelism, if a scenario hangs or the process crashes, there’s no output showing which scenario was in progress.

### Issue Context
Acceptance tests run with `Concurrency: runtime.NumCPU()`, so multiple scenarios can execute concurrently and diagnosing the stuck scenario relies on having a start marker.

### Fix Focus Areas
- acceptance/acceptance_test.go[131-147]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Advisory comments

3. Strict mode hard-coded 🐞 Bug ⛯ Reliability
Description
Strict mode is now always enabled via Strict: true with no way to override (e.g., for local
experimentation). This is likely intentional for CI quality, but consider a flag/env var if
developers need an escape hatch.
Code

acceptance/acceptance_test.go[200]

+		Strict:         true,
Evidence
The test suite hard-codes strict mode on. The acceptance README documents supported -args flags
but does not mention a strictness override.

acceptance/acceptance_test.go[191-201]
acceptance/README.md[29-41]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`godog.Options.Strict` is hard-coded to `true`. This is fine for CI, but if local developers need to temporarily run with incomplete steps, there is no supported override.

### Issue Context
The acceptance test harness already exposes several `-args` flags (`-persist`, `-restore`, `-no-colors`, `-tags`, `-seed`).

### Fix Focus Areas
- acceptance/acceptance_test.go[49-63]
- acceptance/acceptance_test.go[191-201]
- acceptance/README.md[29-41]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

ⓘ The new review experience is currently in Beta. Learn more

Grey Divider

Qodo Logo

@codecov
Copy link

codecov bot commented Feb 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
acceptance 55.48% <ø> (ø)
generative 18.59% <ø> (ø)
integration 27.60% <ø> (ø)
unit 68.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@dheerajodha dheerajodha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@st3penta st3penta merged commit 1738d6a into conforma:main Feb 13, 2026
13 checks passed
@st3penta st3penta deleted the strict-acceptance branch February 13, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants