Skip to content

Quickfix to make a CI test occupy less ram#842

Closed
alex9849 wants to merge 3 commits into
mainfrom
actions_windows_offender
Closed

Quickfix to make a CI test occupy less ram#842
alex9849 wants to merge 3 commits into
mainfrom
actions_windows_offender

Conversation

@alex9849
Copy link
Copy Markdown
Collaborator

@alex9849 alex9849 commented Feb 27, 2026

Quick fix in order to make all CI tests running on Windows.
Set max_repetition_rate and max_nodes_rate to 0 in order to make the test occupy less memory.

Run tests in parallel using max 2 processes.
@github-actions
Copy link
Copy Markdown

ghost commented Feb 27, 2026

🐰 Bencher Report

Branchactions_windows_offender
Testbedubuntu-latest

🚨 2 Alerts

BenchmarkMeasure
Units
ViewBenchmark Result
(Result Δ%)
Upper Boundary
(Limit %)
tests/test_benchmarks.py::test_init_fandangoLatency
milliseconds (ms)
📈 plot
🚷 threshold
🚨 alert (🔔)
162.62 ms
(+8.90%)Baseline: 149.32 ms
159.64 ms
(101.87%)

tests/test_benchmarks.py::test_parse_specLatency
milliseconds (ms)
📈 plot
🚷 threshold
🚨 alert (🔔)
163.37 ms
(+7.33%)Baseline: 152.21 ms
159.31 ms
(102.55%)

Click to view all benchmark results
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)
Upper Boundary
milliseconds (ms)
(Limit %)
tests/test_benchmarks.py::test_generate_with_single_hard_constraint📈 view plot
🚷 view threshold
1,758.16 ms
(-0.99%)Baseline: 1,775.78 ms
2,124.03 ms
(82.77%)
tests/test_benchmarks.py::test_generate_with_single_soft_constraint📈 view plot
🚷 view threshold
57,831.01 ms
(+13.44%)Baseline: 50,980.93 ms
78,707.00 ms
(73.48%)
tests/test_benchmarks.py::test_init_fandango📈 view plot
🚷 view threshold
🚨 view alert (🔔)
162.62 ms
(+8.90%)Baseline: 149.32 ms
159.64 ms
(101.87%)

tests/test_benchmarks.py::test_parse_spec📈 view plot
🚷 view threshold
🚨 view alert (🔔)
163.37 ms
(+7.33%)Baseline: 152.21 ms
159.31 ms
(102.55%)

🐰 View full continuous benchmarking report in Bencher

@alex9849 alex9849 marked this pull request as ready for review February 27, 2026 19:49
@alex9849 alex9849 requested a review from joszamama February 27, 2026 19:49
Comment thread .github/workflows/python-tests.yml Outdated
- name: Run tests
run: |
pytest -v
pytest -v -n 2
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-n auto is specified as the default in pyproject.toml — you're halving the execution speed with this.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regardless: If you think about changing this kind of thing, please talk to me. I have put significant thought into the testing and CI setup — there are reasons for these values.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can undo this part. I tried out a couple things because the windows ci runner was crashing. We want passing tests for the 1.1.0 release. But the fix has been to limit the max_nodes_rate and max_repetition_rate.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joszamama and I decided to just skip the test for now until we have a better understanding of what the problem actually is.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pytest -v -n 2
pytest -v

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Sorry, I didn't see this soon enough.)

Comment thread tests/test_benchmarks.py
Comment on lines +64 to +66
gen = fan.generate_solutions(
max_generations=max_generations, max_repetition_rate=0.0, max_nodes_rate=0.0
)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we do this?

@riesentoaster
Copy link
Copy Markdown
Collaborator

riesentoaster commented Feb 27, 2026

I added an issue for the broken test in #845. If we find a solution for this, we can then close this issue. Do you believe the max_repetition_rate and max_node_rate changes fix the test? If yes: Any idea why?

@alex9849 alex9849 changed the title Enable parallel test execution with pytest Quickfix to make a CI test occupy less ram Feb 28, 2026
@riesentoaster riesentoaster deleted the actions_windows_offender branch May 7, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants