Pull from Mercurial repositories using SSH

[mmuetzel]

Pulling from the remote Mercurial repositories using HTTPS is getting more and more unreliable. It looks like bad actors (maybe AI scrapers, maybe others) are "congesting" ports on the server hosting the repository.
When that happens, pulling using SSH is still working reasonably fast.

@jweaton is in the process of transitioning more and more interactions with the repositories to use SSH. For this, he created users that have read-only access to these repositories.

The buildbot master and the workers are currently interacting with the MXE Octave repository (on the Digital Ocean droplet) using HTTPS. This master and these workers are running inside docker containers.

@siko1056: I hope it is ok to ping you with this.
Do you know if it is possible to have these docker containers establish SSH connections using the key-pair of the read-only users that @jweaton created? Ideally (or maybe that is a requirement), the private part of the key-pair of these users should not be leaked (here or anywhere else).
We could probably consider the hosts as "safe" where these docker containers execute. Could a containerized ssh somehow access the private key on the host? Or is there another better solution for this?

[siko1056]

Hello @mmuetzel ,
Thus the overall plan is to move back to the Digitial Ocean droplet hosted Mercurial repositories for both Octave and MXE Octave?

octave-buildbot/master/defaults/master.cfg

Lines 14 to 26 in ee80cd4

	# URL of the Octave Mercurial (hg) repository to build from
	# and to individual revision (changes). The ID is added automatically.
	OCTAVE_GIT_REPO_URL = "https://github.com/gnu-octave/octave"
	OCTAVE_GIT_REPO_CHANGE_URL = OCTAVE_GIT_REPO_URL + "/commit"
	OCTAVE_BRANCH = "stable"
	# URL of the mxe-octave Mercurial (hg) repository to build from.
	MXE_OCTAVE_HG_REPO_URL = "https://hg.octave.org/mxe-octave"
	MXE_OCTAVE_BRANCH_DEFAULT = "default"
	MXE_OCTAVE_BRANCH_RELEASE = "release"
	MXE_OCTAVE_BRANCHES = [MXE_OCTAVE_BRANCH_DEFAULT, MXE_OCTAVE_BRANCH_RELEASE]

What I realize here is that we must additionally add more parameters for the ssh repository links, as the https urls are also used as reference links in the display page.

Yes, it is possible to add an ssh private key to the master container. For the other way round, sending files to the master via rsync (over ssh), the public keys are already added as authorized_keys there: https://github.com/gnu-octave/doc/blob/main/digital-ocean.md#nightlyoctaveorg.

The key should be added on the Digital Ocean droplet in /opt/Buildbot, e.g. repo_pull_key.

Next mounted to the container with this additional parameter at container creation script:

...
--volume $(pwd)/repo_pull_key:/root/.ssh/id_rsa  \  # to avoid .hgrc for configuration
...

Finally you can switch in the repo to use ssh, without disclosing the credentials or keys.

https://docs.buildbot.net/4.3.0/manual/configuration/changesources.html#hgpoller