Bug Tracking with pytest-bug

This project now uses pytest-bug to track known issues and limitations linked to GitHub issues.

GitHub Issues Created

Categories: Upstream = Windows/.NET/UIA limitation (documented, not a wrapper bug); Flaky = environment/timing instability hardened and/or CI-scoped; Wrapper = under investigation as a genuine wrapper bug; Env = environment guard.

Note on #80 vs #81. #81 is a closed duplicate of #80. The in-code markers now reference GH-80 (the canonical open issue); any lingering GH-81 references should be treated as GH-80. | #82 | test_get_control_type Tab not found during setup | test_value_converter.py::test_get_control_type | UIA2 + WinForms | Flaky | | #83 | ListBox select_by_index fails | test_listbox.py::test_select_by_index | UIA3 + WPF | Not reproducible — CI flake (monitor run=True) | | #89 | Notepad tests on Windows 11 (Store app) | Notepad-based tests (test_getter/search/xpath/keyboard) | Windows 11 | Env (guarded via skip_notepad_on_win11) |

Triage policy (Phase 0). Upstream issues are documented and skipped with platform_limitation / skip markers — they are not wrapper defects and will not be "fixed" here. Flaky issues are hardened (retry/wait) and CI-scoped where needed. Wrapper issues (#80, #83) remain xfail with a bug marker until a fix is verified against the full UIA2/UIA3 × WinForms/WPF matrix. The #88 coverage gate is enforced on the full suite (AppVeyor PR/master) — see Road to v1.0.

Marker taxonomy (which marker to use)

pytest-bug's @pytest.mark.bug is whole-test: with the default run=False it skips the test on every matrix combo, and with run=True it treats the whole test as xfail. It cannot express "skip on WinForms but pass on WPF". Choose the marker by failure shape:

Current assignments after the Phase 0 marker-hygiene pass:

• #76 (tree, flaky on CI across all combos): bug(run=True) — replaced the previous inline try/except AssertionError → pytest.xfail.
• #78 (menu Toggle, WinForms only) and #79 (context menu, UIA3+WinForms only): platform_limitation + conditional skip fixture — replaced the previous in-body pytest.skip.
• #75 (combobox, WinForms): platform_limitation + bug + class-level xfail. Restoring real WPF combobox coverage (skip on WinForms only) is deferred to the coverage PR so it can be validated against the full matrix.
• #89 (test_application.py): aligned to skip_notepad_on_win11 (was a windows11 + conditional xfail), which also avoids the Win11 Store-notepad launch hanging until timeout.

Bug-fix pass outcomes (Phase 0)

Investigation results from running each item against the local UIA2/UIA3 × WinForms/WPF matrix:

• #74 (spinner) — root-caused and resolved as an upstream limitation; locator also hardened. Two changes:
• Locator hardened — the WinForms app exposes two ControlType.Spinner controls; the target is now located by ControlType.Spinner + Name == "Spinner" (independent of the unstable AutomationID that caused the original flakiness) with a retry and an explicit Simple-Controls tab selection (tests/test_utilities/elements/winforms_application/simple_controls.py). The retry also swallows the transient REGDB_E_IIDNOTREG COM hiccup, which surfaces as SystemException (outside the FlaUIException tree) and previously escaped an except ElementNotFound.
• Root cause — even with a perfect locator, the WinForms NumericUpDown UIA peer fails to register on this Windows 11 build under the concurrent multi-process matrix: the control is entirely absent from the accessibility tree while its sibling controls (slider, textbox) on the same reachable window are present. focus(), set_foreground() and a physical tab click do not bring it back — there is no element to find. This matches the documented upstream note ("the spinner control does not work with UIA2/WinForms anymore due to bugs in Windows / .NET") and is not a wrapper defect.

Resolution: the get_spinner fixture now skips with a documented reason when the peer is absent (instead of erroring), while still running the real value/increment/decrement assertions wherever the peer is exposed. The class is marked platform_limitation (the bug(run=True) monitor and xfail are removed), so the run is deterministic — green with a clear skip reason. - #80 (find__with_options) — fixed / reclassified, not a wrapper bug. FlaUI's own UIA2 layer raises NotSupportedByFrameworkException (these methods don't exist in UIA2), so UIA2 is skipped via skip_on_uia2 + platform_limitation. The find_all test also used by_class_name("TabControl") (WPF-only); switched to by_control_type(ControlType.Tab) so it passes on UIA3 + WinForms too. Now passes on UIA3 (both frameworks) and skips on UIA2. - #82 (test_get_control_type) and #83 (ListBox select_by_index) — not reproducible*. Both pass on all four matrix combos locally (the conftest tab-navigation retry already mitigates the #82 setup flakiness). Kept as bug(run=True) monitors; close once CI confirms green.

Usage Examples

Query tests by bug ID

# List all tests affected by bugs
uv run pytest --co -m bug -q

# List tests for specific issue
uv run pytest --co -m "bug and GH-76" -q

# Run tests excluding known bugs
uv run pytest -m "not bug" -v

Check bug annotations in code

@pytest.mark.bug(
    id="GH-76",
    url="https://github.com/amruthvvkp/flaui-uiautomation-wrapper/issues/76",
    reason="Tree selection flaky on AppVeyor CI across all combos",
    run=True,  # run and treat as xfail (BUG-PASS / BUG-FAIL) to detect a future fix
)
def test_flaky_feature():
    pass

Generate bug report

# Show all tests with bug markers
uv run pytest --collect-only -m bug --tb=no

# Count tests per issue
uv run pytest --co -m bug -q | Select-String "test_" | Measure-Object

Marker Usage Pattern

Important: Bug markers are additive metadata, not replacements for xfail/skip markers.

Correct Usage

Always use bug markers alongside xfail/skip:

# Whole-test, all-combo flaky under active triage: run=True treats it as xfail
@pytest.mark.bug(
    id="GH-76",
    url="https://github.com/amruthvvkp/flaui-uiautomation-wrapper/issues/76",
    reason="Tree selection flaky on AppVeyor CI",
    run=True,
)
class TestTreeSelection:
    pass

What Bug Markers Provide

• GitHub issue tracking and links
• Query capability: pytest -m bug
• Summary statistics: "Bugs skipped: X"
• Traceability from test → issue → upstream bug report

What Bug Markers Don't Do

• Don't automatically make tests xfail/skip
• Don't change test execution behavior
• Are metadata only

run parameter

The marker accepts a run flag that controls whether the marked test is skipped or executed:

• run=False (default): skip the test, shown as BUG-SKIP.
• run=True: run the test; it is treated as xfail, shown as BUG-PASS if it passes or BUG-FAIL if it fails. Use this sparingly, only when you want to detect that a bug has been fixed upstream.

Output symbols

A summary line such as Bugs skipped: 4 is printed at the end of the run.

Available pytest-bug Options

• --bug-all-skip: Skip all bug-marked tests
• --bug-all-run: Run all bug-marked tests (respects their xfail/skip)
• --bug-pattern=REGEX: Run matching bug tests only
• --bug-no-stats: Disable summary statistics
• --bug-skip-letter / --bug-fail-letter / --bug-pass-letter: customize output symbols
• --bug-skip-word / --bug-fail-word / --bug-pass-word: customize verbose output words

Customizing defaults in pyproject.toml

[tool.pytest.ini_options]
markers = ["bug: Known bug tracked in GitHub issues"]

# Optional pytest-bug overrides
bug_summary_stats = true
bug_skip_letter = "b"
bug_fail_letter = "f"
bug_pass_letter = "p"
bug_skip_word = "BUG-SKIP"
bug_fail_word = "BUG-FAIL"
bug_pass_word = "BUG-PASS"

Benefits

1. Traceability: Each failing/flaky test is linked to a GitHub issue
2. Queryable: Use -m bug to filter tests by known issues
3. Documentation: Issue URLs provide context and discussion
4. Prioritization: See which bugs affect the most tests
5. CI Integration: Can fail builds only for non-bug failures

Test Status Summary (from AppVeyor)

• Total Tests: 502
• Passed: 423 (84.3%)
• Skipped: 54 (10.8%) - Most due to known bugs
• Xfailed: 12 (2.4%) - Flaky tests
• Tests with Bug Markers: 40 (7.9%)

Next Steps

1. Monitor GitHub issues for upstream fixes (Windows/.NET, FlaUI)
2. Retest when issues are resolved and remove markers
3. Add bug markers to any new flaky/broken tests discovered
4. Use pytest-bug metadata in CI reporting

Related Configuration

• pyproject.toml: Added pytest-bug>=1.4.0 to [dependency-groups] unit-test
• Markers: Added bug marker to pytest configuration
• CI: AppVeyor already captures xfail/skip reasons in test-results.xml