Optimize VCS storage

[joubu]

Every 6 months the Koha community rolls a new release. It's this time of the year and 25.11 has been released and the new version is added to our Weblate instance.

Our Weblate setup is structured as follows:
project: koha
category: versions (main, 25.11, 25.05, etc.)
component: Each version has the same 10 components, each linked to a .po file

The first component is linked to our l10n git repo tracking the relevant branch (e.g. koha/main tracks koha-l10n/main, koha/25.11 tracks koha-l10n/25.11, etc.)

The problem is that the koha-l10n repo keeps growing and is now approximately 11GB. Each version in Weblate has a full clone of the repo leading to significant storage usage:

/app/data/vcs/koha$ du -sbh *
11G     22-11
12G     24-05
12G     24-11
12G     25-05
12G     25-11
11G     base-terminology
11G     main
11G     terminology

Additionally Weblate's DB volumes is 46GB.

To add new versions, I have to remove older ones due to storage constraints.

Is there a way to prevent Weblate from cloning the entire repository for each version? Could this be implemented as a feature, or is there an existing solution I might have missed in the documentation?

Could we have a shallow clone with minimal depth? Or using worktrees?

[nijel]

Weblate does shallow clones by default and fetches a single branch only; see https://docs.weblate.org/en/latest/admin/config.html#vcs-clone-depth.

But it will automatically unshallow when pushing to upstream fails because of the shallow clone. I'm not sure if anything changed at GitLab, but AFAIK it didn't work in the past.

[joubu]

Thanks for your reply!

I wanted to try and confirm what you said, and maybe also try to enable receive.shallowUpdate on a local Forgejo instance to see if it fixes the problem.

However it is not what I am experiencing: the repo is unshallowed before anything is pushed.
When I create a new component there is a shallow clone command (git clone --depth 1 --branch) triggered to scan the repo and list the .po files. After I selected the file mask and saved the form to create the component then I see the size of the repo growing again (watching du -sbh */* in app/data/vcs)

I have tried to debug a bit and added LOGGER.log(logging.DEBUG, args) to Repository.execute but I don't see the fetch --unshallow call from GitRepository.unshallow.
It's possible that I have not restarted correctly all the services at it may be triggered by a background task.

Tried with and without a "Repository push URL".
Also tried with GitLab and GitHub.

Edit: additional info
Actually the repo is still a shallow clone (.git/shallow exists), but during the step "Component is being updated… updating repository" there is a git index-pack command running that is certainly causing the repo to grow.

|   \_ /app/venv/bin/python3 /app/venv/bin/celery worker --hostname celery@%h --loglevel info --queues=celery --pool=threads --prefetch-multiplier=4 --concurrency 4
|   \_ git fetch origin
|       \_ /usr/lib/git-core/git remote-https origin https://github.com/joubu/koha-l10n.git
|   \_ /usr/lib/git-core/git-remote-https origin https://github.com/joubu/koha-l10n.git
|       \_ /usr/lib/git-core/git index-pack --stdin --fix-thin --keep=fetch-pack 1445 on 602c21428b04 --pack_header=2,416310

[nijel]

Maybe the repository is still shallow, but other branches are causing that? Check with git rev-parse --is-shallow-repository.

[nijel]

In my tests this seems to be the problem - the original clone is shallow, but next fetch fetches all the branches without specifying depth. So the checked out branch is shallow, but others have full history. #17276 should fix this.

[joubu]

In my tests this seems to be the problem - the original clone is shallow, but next fetch fetches all the branches without specifying depth. So the checked out branch is shallow, but others have full history. #17276 should fix this.

Yes, it's definitely better. Now it goes from 653M to 852M. There is still "something" happening (I guess it's git internal doing stuffs with the objects). But we are far from the 11GB!
Thanks!

I will keep investigating and try to push some translations now.

[nijel]

That's probably the size of additional branches which fetch downloads.

[joubu]

In testing env:
I do confirm that the shallow clone repo was able to push to gitlab and so didn't unshallow!
The repo size stays the same.

In production:
I was able to remove some refs using the following commands:
git show-ref to list all the tracked branches
git update-ref -d $ref to deref the branches
git repack -adf --depth=50 --window=50 repack
git's size-pack is now 58M and size of the repo (including .git) is 655M (vs 12G before)

However the prod server is not using the dev branch yet so I expect it to refetch the branches again.

Thanks again for your work on this @nijel!

[nijel]

Yes, it will again fetch all the remotes until you upgrade (the release is scheduled on Monday).

Also, the Git maintenance is something I wanted to add and forgot to create an issue for; #17288 now tracks that.