[BUG] Windows CI build is failing

[lohedges]

The Windows CI build of SIre has inexplicably started failing. This was working for the last PR, but failed on the merge to devel. (Same code.) The failure seems to occur at the end of the wrapper compilation, at the point that Sire is about to be installed into the environment.

Here is the output from the end of the failed build:

2024-12-14T09:18:04.9464215Z Traceback (most recent call last):
2024-12-14T09:18:04.9466400Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996: 'stdext::checked_array_iterator<const T *>::_Unwrapped': warning STL4043: stdext::checked_array_iterator, stdext::unchecked_array>
2024-12-14T09:18:04.9469146Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996:         with [%SRC_DIR%\build\conda_build_wrapper\Mol\Mol.vcxproj]
2024-12-14T09:18:04.9470277Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996:         [ [%SRC_DIR%\build\conda_build_wrapper\Mol\Mol.vcxproj]
2024-12-14T09:18:04.9471410Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996:             T=qint64 [%SRC_DIR%\build\conda_build_wrapper\Mol\Mol.vcxproj]
2024-12-14T09:18:04.9472540Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996:         ] [%SRC_DIR%\build\conda_build_wrapper\Mol\Mol.vcxproj]
2024-12-14T09:18:04.9473414Z   (compiling source file '../../../wrapper/Mol/SireMol_registrars.cpp')
2024-12-14T09:18:04.9473737Z
2024-12-14T09:18:04.9473896Z   Generating Code...
2024-12-14T09:18:04.9475005Z      Creating library C:/Users/runneradmin/miniconda3/envs/sire_build/conda-bld/sire_1734161070384/work/build/conda_build_wrapper/Mol/Release/Mol.lib and object C:/Users/runneradmin/miniconda3/envs/sire_build/conda-bld/sire_1734161070384/work/build/conda>
2024-12-14T09:18:04.9476345Z   LINK : /LTCG specified but no code generation required; remove /LTCG from the link command line to improve linker performance
2024-12-14T09:18:04.9476901Z   Mol.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Mol\Release\_Mol.pyd
2024-12-14T09:18:04.9477240Z SOMETHING WENT WRONG WHEN COMPILING THE WRAPPERS!
2024-12-14T09:18:04.9477706Z   File "C:\Users\runneradmin\miniconda3\envs\sire_build\Scripts\conda-build-script.py", line 10, in <module>
2024-12-14T09:18:04.9525638Z     sys.exit(execute())
2024-12-14T09:18:04.9534216Z              ^^^^^^^^^
2024-12-14T09:18:04.9537416Z   File "C:\Users\runneradmin\miniconda3\envs\sire_build\Lib\site-packages\conda_build\cli\main_build.py", line 590, in execute
2024-12-14T09:18:04.9579976Z     api.build(
2024-12-14T09:18:04.9594295Z   File "C:\Users\runneradmin\miniconda3\envs\sire_build\Lib\site-packages\conda_build\api.py", line 250, in build
2024-12-14T09:18:04.9605286Z     return build_tree(
2024-12-14T09:18:04.9605580Z            ^^^^^^^^^^^
2024-12-14T09:18:04.9611521Z   File "C:\Users\runneradmin\miniconda3\envs\sire_build\Lib\site-packages\conda_build\build.py", line 3638, in build_tree
2024-12-14T09:18:04.9651585Z     packages_from_this = build(
2024-12-14T09:18:04.9651949Z                          ^^^^^^
2024-12-14T09:18:04.9655274Z   File "C:\Users\runneradmin\miniconda3\envs\sire_build\Lib\site-packages\conda_build\build.py", line 2462, in build
2024-12-14T09:18:04.9660301Z     windows.build(
2024-12-14T09:18:04.9666484Z   File "C:\Users\runneradmin\miniconda3\envs\sire_build\Lib\site-packages\conda_build\windows.py", line 339, in build
2024-12-14T09:18:04.9683414Z     check_call_env(
2024-12-14T09:18:04.9684001Z   File "C:\Users\runneradmin\miniconda3\envs\sire_build\Lib\site-packages\conda_build\utils.py", line 405, in check_call_env
2024-12-14T09:18:04.9695952Z     return _func_defaulting_env_to_os_environ("call", *popenargs, **kwargs)
2024-12-14T09:18:04.9711468Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-12-14T09:18:04.9715388Z   File "C:\Users\runneradmin\miniconda3\envs\sire_build\Lib\site-packages\conda_build\utils.py", line 381, in _func_defaulting_env_to_os_environ
2024-12-14T09:18:04.9717125Z     raise subprocess.CalledProcessError(proc.returncode, _args)
2024-12-14T09:18:04.9735048Z subprocess.CalledProcessError: Command '['cmd.exe', '/d', '/c', 'conda_build.bat']' returned non-zero exit status 4294967295.
2024-12-14T09:18:20.4380116Z ##[error]Process completed with exit code 1.
2024-12-14T09:18:20.4720794Z Cleaning up orphan processes
2024-12-14T09:18:20.6298207Z Terminate orphan process: pid (4696) (vctip)
2024-12-14T09:18:20.6412216Z Terminate orphan process: pid (2132) (MSBuild)
2024-12-14T09:18:20.6606559Z Terminate orphan process: pid (2808) (conhost)

And the same point in the last working build:

2024-12-09T16:33:11.7124678Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996: 'stdext::checked_array_iterator<const T *>::_Unwrapped': warning STL4043: stdext::checked_array_iterator, stdext::unchecked_array>
2024-12-09T16:33:11.7126778Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996:         with [%SRC_DIR%\build\conda_build_wrapper\Mol\Mol.vcxproj]
2024-12-09T16:33:11.7127778Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996:         [ [%SRC_DIR%\build\conda_build_wrapper\Mol\Mol.vcxproj]
2024-12-09T16:33:11.7128893Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996:             T=qint64 [%SRC_DIR%\build\conda_build_wrapper\Mol\Mol.vcxproj]
2024-12-09T16:33:11.7129864Z C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.42.34433\include\xutility(1406,42): warning C4996:         ] [%SRC_DIR%\build\conda_build_wrapper\Mol\Mol.vcxproj]
2024-12-09T16:33:11.7130558Z   (compiling source file '../../../wrapper/Mol/SireMol_registrars.cpp')
2024-12-09T16:33:53.2839753Z
2024-12-09T16:33:53.2840252Z   Generating Code...
2024-12-09T16:33:53.2841345Z      Creating library C:/Users/runneradmin/miniconda3/envs/sire_build/conda-bld/sire_1733755352756/work/build/conda_build_wrapper/Mol/Release/Mol.lib and object C:/Users/runneradmin/miniconda3/envs/sire_build/conda-bld/sire_1733755352756/work/build/conda>
2024-12-09T16:33:53.2842701Z   LINK : /LTCG specified but no code generation required; remove /LTCG from the link command line to improve linker performance
2024-12-09T16:33:53.2843263Z   Mol.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Mol\Release\_Mol.pyd
2024-12-09T16:33:53.2843846Z   Building Custom Rule C:/Users/runneradmin/miniconda3/envs/sire_build/conda-bld/sire_1733755352756/work/wrapper/CMakeLists.txt
2024-12-09T16:33:53.2844265Z
2024-12-09T16:33:53.2844329Z Installing sire
2024-12-09T16:33:53.2844763Z NOW RUNNING "%BUILD_PREFIX%\Library\bin\cmake.exe" --build . --target INSTALL -- /m:2 /p:Configuration=Release /p:Platform=x64
2024-12-09T16:33:53.2845249Z MSBuild version 17.12.6+db5f6012c for .NET Framework
2024-12-09T16:33:53.2845434Z
2024-12-09T16:33:53.2845641Z   SirePython.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Helpers\Release\SirePython.dll
2024-12-09T16:33:53.2846092Z   Base.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Base\Release\_Base.pyd
2024-12-09T16:33:53.2846497Z   Maths.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Maths\Release\_Maths.pyd
2024-12-09T16:33:53.2846886Z   MM.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\MM\Release\_MM.pyd
2024-12-09T16:33:53.2847246Z   IO.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\IO\Release\_IO.pyd
2024-12-09T16:33:53.2847587Z   ID.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\ID\Release\_ID.pyd
2024-12-09T16:33:53.2847922Z   FF.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\FF\Release\_FF.pyd
2024-12-09T16:33:53.2848295Z   Error.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Error\Release\_Error.pyd
2024-12-09T16:33:53.2848721Z   Cluster.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Cluster\Release\_Cluster.pyd
2024-12-09T16:33:53.2849139Z   Mol.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Mol\Release\_Mol.pyd
2024-12-09T16:33:53.2849501Z   CAS.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\CAS\Release\_CAS.pyd
2024-12-09T16:33:53.2849915Z   Analysis.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Analysis\Release\_Analysis.pyd
2024-12-09T16:33:53.2850316Z   Qt.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Qt\Release\_Qt.pyd
2024-12-09T16:33:53.2850681Z   Move.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Move\Release\_Move.pyd
2024-12-09T16:33:53.2851087Z   Search.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Search\Release\_Search.pyd
2024-12-09T16:33:53.2851570Z   SireGemmi.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Convert\SireGemmi\Release\_SireGemmi.pyd
2024-12-09T16:33:53.2852102Z   SireRDKit.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Convert\SireRDKit\Release\_SireRDKit.pyd
2024-12-09T16:33:53.2852645Z   SireOpenMM.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Convert\SireOpenMM\Release\_SireOpenMM.pyd
2024-12-09T16:33:53.2853739Z   Stream.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Stream\Release\_Stream.pyd
2024-12-09T16:33:53.2854156Z   Squire.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Squire\Release\_Squire.pyd
2024-12-09T16:33:53.2854570Z   Units.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Units\Release\_Units.pyd
2024-12-09T16:33:53.2855067Z   System.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\System\Release\_System.pyd
2024-12-09T16:33:53.2855579Z   sire_python.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\python\Release\sire_python.exe
2024-12-09T16:33:53.2856003Z   Vol.vcxproj -> %SRC_DIR%\build\conda_build_wrapper\Vol\Release\_Vol.pyd

The error message states that SOMETHING WENT WRONG WHEN COMPILING THE WRAPPERS!, however I don't see any exceptions or error messages when scrolling back.

Looking at the runner setup for the two builds I see:

Failed:

Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Runner Image
  Image: windows-2022
  Version: 20241211.1.0
  Included Software: https://github.com/actions/runner-images/blob/win22/20241211.1/images/windows/Windows2022-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/win22%2F20241211.1
Runner Image Provisioner
  2.0.385.1

Working:

Current runner version: '2.321.0'
Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Runner Image
  Image: windows-2022
  Version: 20241201.2.0
  Included Software: https://github.com/actions/runner-images/blob/win22/20241201.2/images/windows/Windows2022-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/win22%2F20241201.2
Runner Image Provisioner
  2.0.385.1

The only difference is the runner image version, i.e. 20241201.2.0 for the working build, and 20241211.1.0 for the failing one.

Given the lack of any message in the output, I'm wondering if it's a memory issue.

Unfortunately this means that we don't currently have a working version of somd for any recent windows dev packages or releases. A user would still be able to build from source, though.

[chryswoods]

Yes, that does look like a memory or out of disk space issue. Likely memory as it failed compiling SireMol. A fix could be to reduce the number of workers compiling the python wrappers (I think it was the --npycores option to the build scripts, or a similar environment variable if memory serves)

[lohedges]

I had a quick check and there doesn't appear to have been a change in the hardware provisioning for the image. For windows-latest, there should still be 4 CPUS, 4 GB RAM, and 14GB storage. I'll try reducing npycores in the Windows build script to see if that fixes things.

[lohedges]

I tried using --npycores 1 for the Windows build and it still failed. Not sure what other options there are.