[Tooling] Port of generate_tests.py to Julia

[BethanyG]

ooof. This was less of an exercise in programming and more an Epic Yak Shaving experience^(TM)

Still not totally confident in everything, hence the "draft" status. I am also displeased with the performance, which is slowed considerably by juliacall. Will experiment with alternatives, but I don't think precompiling the formatter is exactly the way we want to go. But we might want to try using Python's subprocess module to call Julia directly and invoke the Formatter. My "googling" said that would be slower ... but as it stands it feels slow to me. So.

We may also need to add more functions/macros/filters as we make more complicated templates.

I am also not confident that the Formatter is really adding value. Despite passing in what I think are the correct arguments, newlines are not always stripped. I resorted to JinJa foo to remove newlines, and so I can't really say that the formatter is going to format the way we want. Sigh. Definitely a WIP.

Added the following scripts/files:

• bin/data.py
• bin/githelp.py
• bin/generate_tests.py (modified for Julia and JuliaFormatter & using Juliacall)
• config/generator_macros.j2 (modified for Julia)
• exercises/practice/acronym/.meta/template.j2 (test template for acronym)
• exercises/practice/acronym/runtests.jl (generated test file for acronym)
• requirements.txt (Python requirements for the generator)
• Project.tom (didn't know where else to put JuliaFormatter, so it is here but commented out)
• .JuliaFormatter.toml (config file, but I also pass YASStyle as an argument to juliacall in the generator script)

This was an irritating experience. I ran the tests from my Exercism Python conda env, and despite Julia and JuliaFormatter being on my path, juliacall saw fit to install its own Julia into the env. Total waste of disk space. 😡

Open to suggestions around JuliaFormatter, juliacall, and anything else you can think of. Oh - -and this uses Python 3.13.5, although you can probably use 3.14.

[github-actions]

This PR touches files which potentially affect the outcome of the tests of an exercise. This will cause all students' solutions to affected exercises to be re-tested.

If this PR does not affect the result of the test (or, for example, adds an edge case that is not worth rerunning all tests for), please add the following to the merge-commit message which will stops student's tests from re-running. Please copy-paste to avoid typos.

[no important files changed]

For more information, refer to the documentation. If you are unsure whether to add the message or not, please ping @exercism/maintainers-admin in a comment. Thank you!

[colinleach]

This sounds like we inflicted unintended pain on you. Apologies!

I'm pretty frazzled at this point, so defusing the bomb will need to wait for tomorrow: another day, hopefully more awake. Please don't expect miracles...

[BethanyG]

This sounds like we inflicted unintended pain on you. Apologies!

No worries. Now that things are set up, I expect a whole lot less drama. But VSCode and I are NOT friends right now.

[colinleach]

Oof, reviewing this properly is going to take a while, and a clearer head than I currently have. All in good time...

Some initial observations (beyond the obvious "thank you for doing this"):

• I think you may be worrying too much about performance. This code isn't run often enough to justify a big tuning effort: some initial conversions of existing exercises, occasional dependabot stuff, addition of some more practice exercises. Getting it right is much more important than making it fast.
• I've not used JuliaCall and barely knew it existed. My guess is that PyCall is much more developed, because Julia depends on Python libraries much more than Python depends on Julia. Age and maturity, backed by some explicit policy ("why would we devote resources to rewrite BS4 when we can just call it from Python?").
• Trivial and totally non-urgent, but at some point we need to remember to remove comments about air.

I'm tempted to suggest that we approve and merge this fairly quickly, then keep kicking the tyres/tires to really get a feel for it. None of this is student-facing.

[colinleach]

I haven't explored this yet, but I find myself wondering how reproducible the slowness is. In Julia, the first call to any function hits a delay for JIT-compilation (serious, optimizing-compiler stuff). Subsequent calls should be much faster, though I don't know how that plays with Python.

[BethanyG]

Updated the air refs (😊). Putting it in "ready to review" status. I can always experiment with Python subprocess on a different branch.

Oof, reviewing this properly is going to take a while, and a clearer head than I currently have.

FWIW, this is the same code the Python track uses and the R track uses ..... with the change of juliacall, JuliaFormatter, and JinJa macros modification. Yes - that's still a lot (more than I had to do for R), but you don't need to go over the Python code that slices and dices problem-specs, if that helps.

[colinleach]

I'm still getting ERROR:generator:the JuliaFormatter utility must be installed when I try to run generate_tests.py

It's installed in my system Julia, it's installed through juliapkg in my .venv, yet it's not being found.

Any other suggestions?

[colinleach]

I ran the tests from my Exercism Python conda env, and despite Julia and JuliaFormatter being on my path, juliacall saw fit to install its own Julia into the env.

FWIW, my .venv seems happy to find and use ~/.juliaup/bin/julia like everything else. It's just a pity it can't see the installed JuliaFormatter package.

As you suggested, this is very off-pissing.

[colinleach]

Current workaround is to install jlfmt and run it from the command line. Admittedly, this is some distance from the Python integration we want.

[BethanyG]

AARGH. So sorry this is getting you too. Very frustrating. I am trying to think what I did for JuliaFormatter that might be different. Lemme check my paths...perhaps I added something there and then forgot? I was able to have things called fine from the script, so something is different.... 🤔

[BethanyG]

Hummmm....I think I introduced a bug. I am getting the same error you are! 😱

sigh. Hold, plz while I un-F*&k the code.......

[colinleach]

No rush. My old, tired head is telling me I'm done for the day.

[colinleach]

Just a question (this is not my expertise): would shelling out from Python to the command line to run jlfmt be significantly worse than all the juliacall stuff? Maybe simpler, but it I guess it might really mess with error reporting?

[BethanyG]

Just a question (this is not my expertise): would shelling out from Python to the command line to run jlfmt be significantly worse than all the juliacall stuff? Maybe simpler, but it I guess it might really mess with error reporting?

I would have to try it using subprocess. From what I can read, it has the effect of all the "startup" stuff for Julia every time you do it (plus compliation). However, I am no expert here - the proof would be in actually testing it.

BTW: fixed the issue. I will push an update here in a sec. I had named the check for the app "JuliaFormatter", but it turns out it needs to be "jlfmt" -- since that's the app that has the CLI. So the CLI as well as the lib need to be installed. Sorry about that!

[BethanyG]

I have to add that after this ride on the struggle bus, I find it a tad offensive that JuliaFormatter pats itself on the back:

with emoji no less!

[colinleach]

Nope, I'm still getting the same error.

Something that will need to wait till tomorrow.

I'll approve it meanwhile, without merging. I'd like to give @depial a chance to comment (if he's around).

[BethanyG]

Shelling out works and it's not slower. But it is much simpler - so expect an update here shortly. Extra bonus because we wouldn't need a config file at all this way - we just need to figure out the options we want to pass on to the formatter CLI! 🎉

The only thing that's now bugging me is that the newline removal is really aggressive - so we'll have to tune it probably.

[colinleach]

Provisionally approved, accepting that there are still some formatting tweaks to sort out.

[BethanyG]

Nope, I'm still getting the same error.

That is seriously weird. Can't figure out what might be the difference. I am going to push an update to do the shell out instead (but I'll leave the Juliacall code in commented out). Let's not merge until we figure it out.

[colinleach]

I think it's a $PATH problem at my end. My bashrc adds ~/.julia/bin as this contains jlfmt, but the .venv is not seeing it (but is seeing .juiliaup/bin, which contains the Julia starter).

All fixable, I'm sure. Just not today.

[BethanyG]

I think it's a $PATH problem at my end. My bashrc adds ~/.julia/bin as this contains jlfmt, but the .venv is not seeing it (but is seeing .juiliaup/bin, which contains the Julia starter).

All fixable, I'm sure. Just not today.

I am using .bash_profile, as well as bashrc and .profile (which I think might be a mac thing). Anyways, juliaup wrote into all three. So ... it might be a difference between a login shell and a session shell? Not sure. Good speed tho, and I hope you feel better and get some good rest!

I should be around tomm, should you find more bugs. 😄

[depial]

Sorry... It's getting down to crunch time for me here, so I'm not likely to have much time to review thoroughly. Let me see how much I can get done on what I'm working on today, and I'll see if I can start exploring soon. Of course, since you feel ready to merge, feel free to do so. I agree we can work out any details/bugs as we go.

In the meantime, I did take a quick look and noticed:

• I see the template is in line with the requirements for verbose testing (w/ the wrapping testset macro), so that's a go.
• There's less white space than I'd expect (e.g. no newline after a testset), but I don't really mind (there's also the GitHub complaint about no new lines at the end of some files, which I also don't really care about).
• I'm unsure how we will deal with auxiliary files, but alphametics is really a one-off and we should only use aux files with custom/concept exercises, rather than canonical exercises, so this concern can be ignored.

[colinleach]

It's getting down to crunch time for me here

Good luck!

There's less white space than I'd expect

Yes, kind of a known problem. We need to think how we might improve it, but meanwhile it's not a deal-breaker.

When I get chance, I'll restart a few things on my machine and see if I can get it working (with everything findable on a suitable path). Given that, I hope to merge later today.

[colinleach]

Working now, I'm happy to say!

I'll press the Big Green Button once @BethanyG gives the OK (unless Erik manages to add maintainer rights before then).

[colinleach]

Tuning the formatter might be the cleanest solution, but if that goes badly one fallback is a quick regex replace. For unnested tests, we just need a newline added after end and before an indented @testset. I'd need to play more to see how this plays with nesting - first impressions are it's OK.

[BethanyG]

Tuning the formatter might be the cleanest solution, but if that goes badly one fallback is a quick regex replace.

I think we can also turn off the newline removal in JuliaFormatter by passing remove_extra_newlines=false. But then we need to be really careful about the formatting codes we use with JinJa. That's pretty frustrating, but we could probably do a master template (or series) that we'd only have to think through once. YMMV.

[colinleach]

Should I merge meanwhile, or would you prefer to wait?

[BethanyG]

I have some stuff to do this morning, but can make that change this afternoon. If you'd like to experiment, the line that calls JuliaFormatter is 296 in generate_tests.py. You can either add "remove_extra_newlines=false" as an argument after "-v", or remove "--style=yas"

The template I made for acronym has the newline formatting in it, so that is a starting example of what we can do in JinJa. Be warned tho: its very frustrating/fiddly to work with, but is probably less irritating than getting a regex correct.

[BethanyG]

Should I merge meanwhile, or would you prefer to wait?

Let's wait a couple hours. I want to make sure things are a little tighter, and it's easier for me to push changes before we merge - if that's OK?

[colinleach]

Suggested change: {%- endmacro to {% endmacro to create a blank line between testsets.

[colinleach]

As far as I can tell, "remove_extra_newlines=false" is the default. I got good results by removing one - in the template to actually create the blank lines.

Delightfully simple, if true.