Run screenshot tests on macOS and Windows CI#2636
Draft
Conversation
- Add ScreenshotTestsMacOS job (macos-latest) to main.yml - Add ScreenshotTestsWindows job (windows-latest) with Mesa3D setup to main.yml - Update build.gradle to add -XstartOnFirstThread on macOS and Mesa3D lib path support - Update TestDriver.java to run app on main thread on macOS (GLFW requirement) - Update screenshot-test-comment.yml to monitor all three platform jobs Co-authored-by: riccardobl <4943530+riccardobl@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Run screenshot tests on Android, Windows, and macOS
Run screenshot tests on macOS and Windows CI
Mar 3, 2026
Member
|
@copilot fails on macos and it seems stuck on windows (can't terminate after the test, somehow) also, please make the linux screenshots the ground truth and compare all the other platforms to linux output, make the test pass if the results are within a reasonable threshold. |
- macOS: change runner to macos-13 (Intel) to avoid NSGL pixel format error on Apple Silicon CI with GLFW 3.4; the Apple OpenGL compat layer on macOS 14+ can fail to find a suitable pixel format in virtualised CI environments - Windows: pre-set onError before app.start() so context-creation failures count down the latch immediately; wrap executor lambda in try/catch for UnsatisfiedLinkError; use app.stop(false) in timeout path to avoid deadlocking if glfwTerminate hangs inside Mesa3D; fix Mesa3D env vars (GALLIUM_DRIVER, MESA_GL_VERSION_OVERRIDE) to persist via GITHUB_ENV; use softpipe driver to avoid slow LLVM JIT startup from llvmpipe - Cross-platform threshold: add jme.screenshotTest.pixelTolerance system property; imagesAreTheSame() now allows per-pixel differences within the tolerance (using existing getMaximumComponentDifference); set tolerance=5 for macOS/Windows in build.gradle so Linux stays the ground truth Co-authored-by: riccardobl <4943530+riccardobl@users.noreply.github.com>
Contributor
Author
Fixed in 8f774d3. Three changes: macOS pixel-format failure: Switched the runner from Windows stuck: Multiple fixes:
Linux as ground truth / threshold comparison: Added |
|
🖼️ Screenshot tests have failed. The purpose of these tests is to ensure that changes introduced in this PR don't break visual features. They are visual unit tests. 📄 Where to find the report:
✅ If you did mean to change things: ✨ If you are creating entirely new tests: Note; it is very important that the committed reference images are created on the build pipeline, locally created images are not reliable. Similarly tests will fail locally but you can look at the report to check they are "visually similar". See https://github.com/jMonkeyEngine/jmonkeyengine/blob/master/jme3-screenshot-tests/README.md for more information Contact @richardTingle (aka richtea) for guidance if required |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Screenshot tests previously only ran on Linux (Ubuntu + OpenGL Docker container). This adds macOS and Windows CI jobs so visual regressions are caught across platforms.
Changes
CI (
main.yml)ScreenshotTestsMacOS— new job onmacos-13(Intel x86_64) using Java 21; artifacts asscreenshot-test-report-macos. Usesmacos-13rather thanmacos-latestbecause GLFW 3.4's NSGL backend can fail with "NSGL: Failed to find a suitable pixel format" on Apple Silicon CI environments where Apple's OpenGL compatibility layer has restrictions in virtualised setups.ScreenshotTestsWindows— new job onwindows-latestusing Java 21; downloads Mesa3Dpal1000/mesa-dist-winv24.2.4 for software OpenGL (no hardware GPU on GH Actions Windows runners); usesGALLIUM_DRIVER=softpipeto avoid slow LLVM JIT startup; all Mesa env vars (GALLIUM_DRIVER,MESA_GL_VERSION_OVERRIDE,MESA_GLSL_VERSION_OVERRIDE) are persisted viaGITHUB_ENVso they are available to the test step; artifacts asscreenshot-test-report-windowsjme3-screenshot-tests/build.gradle-XstartOnFirstThreadJVM arg on macOS — GLFW/LWJGL3 requires all window operations on the main OS threadMESA3D_LIBS_PATHenv var and forwards it asorg.lwjgl.librarypathsystem property so LWJGL3 extracts its native DLLs into the same directory as Mesa'sopengl32.dll; Windows DLL search order then picks up Mesa's software renderer over the system stubjme.screenshotTest.pixelTolerance=5on macOS and Windows so that minor per-pixel rendering differences (GPU driver rounding) don't cause spurious failuresTestDriver.javaOn macOS with
-XstartOnFirstThread, JUnit runs on the macOS main thread.app.start()is now called directly (blocking) on the current thread instead of dispatching to an executor, satisfying GLFW's main-thread requirement.Additional robustness improvements for all non-Linux platforms:
App.onErroris pre-set inbootAppForTest()beforeapp.start()so that context-creation failures (which happen beforeTestDriver.initialize()runs) immediately count down the latch instead of leaving the test thread blocked for the full timeoutUnsatisfiedLinkErrorand similar exceptions that would otherwise swallow startup failures silentlyapp.stop(true)in the timeout path is replaced withapp.stop(false)(non-blocking) to prevent the test thread from deadlocking ifglfwTerminate()hangs inside Mesa3DPIXEL_TOLERANCEconstant read fromjme.screenshotTest.pixelTolerancesystem property;imagesAreTheSame()usesgetMaximumComponentDifference()per pixel when tolerance > 0, allowing minor cross-platform rendering differences while still catching real visual regressionsscreenshot-test-comment.ymlNotes
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.