Skip to content

Merge ssh-host-(re)sign and add to all.yml #1827

Open
mira-miracoli wants to merge 14 commits intousegalaxy-eu:masterfrom
mira-miracoli:fix-ssh-sign
Open

Merge ssh-host-(re)sign and add to all.yml #1827
mira-miracoli wants to merge 14 commits intousegalaxy-eu:masterfrom
mira-miracoli:fix-ssh-sign

Conversation

@mira-miracoli
Copy link
Copy Markdown
Contributor

This should check if the ssh host certs will expire in less then a month and if so or if the certs are missing, sign them.

The role is added to all.yml playbook.

@mira-miracoli mira-miracoli requested a review from kysrpex January 14, 2026 12:22
all.yml Outdated
roles:
- role: usegalaxy_eu.ssh_manager
- ssh-host-sign
- dev-sec.ssh-hardening
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good, although there are more playbooks that include dev-sec.ssh-hardening than playbooks that do not include it.

I'll run this role in the all.yml playbook with --diff --check to ensure nothing undesired would happen to the SSH config of those servers.

Copy link
Copy Markdown
Contributor

@kysrpex kysrpex Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TL;DR We have to set the host var ssh_server_ports: ["8080"] for apps.galaxyproject.eu and see what's going on with policycoreutils-python, then check again.


I ran all.yml (with every role comment out but dev-sec.ssh-hardening) with --diff --check.

all.yml

---
- name: Manage authorized SSH keys
  hosts: all
  gather_facts: true
  roles:
    # - role: usegalaxy_eu.ssh_manager
    # - ssh-host-sign
    - role: dev-sec.ssh-hardening
      become: true

I was not able to run the playbook until the end for all hosts because of this error fatal: [celery-1.galaxyproject.eu]: FAILED! => {"changed": false, "failures": ["No package policycoreutils-python available."], "msg": "Failed to install some of the specified packages", "rc": 1, "results": []} related to selinux dependencies, but it ran nevertheless almost until the end.

There are really a lot of changes, most are inconsequential (creating an empty file or changing some permissions). The task TASK [dev-sec.ssh-hardening : create sshd_config and set permissions to root/600] is critical though, because it changes the sshd configuration.

The log from running the playbook is really long, so are the diffs. For that task alone it's about 5k lines. Therefore I used an LLM to do this "simple" but very tedious and long task for me.

Prompt
# TASK: Analyze changes to the OpenSSH daemon configuration file

I am going to share with you an Ansible playbook execution log. This playbook applies to several hosts. One of the tasks modifies the sshd configuration `TASK [dev-sec.ssh-hardening : create sshd_config and set permissions to root/600] ***`.

Please read the log of this task (ignore all other tasks) and determine the changes for each SSH setting.

Do not do anything until I paste the log.

## Understanding the SSH configuration file

From the Linux manual page for `sshd_config`:

\```
sshd(8) reads configuration data from /etc/ssh/sshd_config (or the file specified with -f on the command line).  The file contains keyword-argument pairs, one per line.  Unless noted otherwise, for each keyword, the first obtained value will be used.  Lines starting with ‘#’ and empty lines are interpreted as comments. Arguments may optionally be enclosed in double quotes (") in order to represent arguments containing spaces.

The possible keywords and their meanings are as follows (note that keywords are case-insensitive and arguments are case-sensitive):
\```

## Interpreting changes

In the execution log, you will read diffs. Diffs highlight changes between an old state and a new state. For each setting you have to determine what was the old value and what is the new value.

Besides the defined values, settings can be also in undefined state and in unknown state (a setting is unknown if it does not appear neither in the old nor new state).

For each state, ignore settings that are present but commented out, those are to all effects undefined.

Match a specific `hostname` with its diff as done in below for `apps.galaxyproject.eu`.

\```
... -> THIS DIFF MACTHES HOSTNAME `traefik.galaxyproject.eu`.


changed: [traefik.galaxyproject.eu]
... -> THIS DIFF MACTHES HOSTNAME `apps.galaxyproject.eu`.


changed: [apps.galaxyproject.eu]
... -> THIS DIFF MATCHES ANOTHER HOST
\```

If you use this rule, then `Port` changes exclusively for one host: `apps.galaxyproject.eu`. Ask yourself "Did `Port` change from 8080 to 22 for `apps.galaxyproject.eu`?" The correct answer is yes.

Consider changes for each host individually. Do not group them.

## Preparing the results

To facilitate your thinking, put the changes you found for all hosts in the form of tables. Each table should identify to which host/group it refers, and include columns for the old setting and the new setting. Each row should represent a setting. If a setting is undefined, use the character `-`. If it's unknown, use the character `x`.

## Showing the results

Show **ONLY A SINGLE TABLE** agglomerating all changes, for each setting, you should show the union (as in mathematics) of all values for both states (new and old) for all hosts.

Result

Keep in mind that in this table is constructed using information that's on the diff. The LLM simply doesn't have access to the complete content of the files.

Also this is an LLM and makes mistakes, but I assume the overview of the variables that have changed is wide enough.

SSH setting Old value(s) observed New value(s) observed
Port 22, 8080, - 22
AddressFamily any, - inet
ListenAddress 0.0.0.0, - 0.0.0.0
PermitRootLogin yes, no, prohibit-password, - without-password
Protocol 2, - 2
HostKey /etc/ssh/ssh_host_rsa_key, /etc/ssh/ssh_host_ecdsa_key, /etc/ssh/ssh_host_ed25519_key, - /etc/ssh/ssh_host_rsa_key, /etc/ssh/ssh_host_ecdsa_key, /etc/ssh/ssh_host_ed25519_key
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub, /etc/ssh/ssh_host_ecdsa_key-cert.pub, /etc/ssh/ssh_host_ed25519_key-cert.pub, - /etc/ssh/ssh_host_rsa_key-cert.pub, /etc/ssh/ssh_host_ecdsa_key-cert.pub, /etc/ssh/ssh_host_ed25519_key-cert.pub
StrictModes yes, - yes
SyslogFacility AUTH, AUTHPRIV, - AUTH
LogLevel INFO, VERBOSE, - VERBOSE
LoginGraceTime 2m, 30s, - 30s
MaxAuthTries 6, 2, - 2
MaxSessions 10, - 10
MaxStartups 10:30:60, 10:30:100, - 10:30:100
PubkeyAuthentication yes, - yes
PasswordAuthentication yes, no, - no
PermitEmptyPasswords no, - no
ChallengeResponseAuthentication yes, no, - no
AuthenticationMethods - publickey
UsePAM yes, no, - yes
IgnoreRhosts yes, - yes
IgnoreUserKnownHosts yes, - yes
HostbasedAuthentication no, - no
KerberosAuthentication yes, no, - no
KerberosOrLocalPasswd yes, no, - no
KerberosTicketCleanup yes, - yes
GSSAPIAuthentication yes, no, - no
GSSAPICleanupCredentials no, yes, - yes
Ciphers - chacha20-poly1305@openssh.com, aes256-gcm@openssh.com, aes128-gcm@openssh.com, aes256-ctr, aes192-ctr, aes128-ctr
MACs hmac-ripemd160, hmac-ripemd160-etm@openssh.com, - hmac-sha2-512-etm@openssh.com, hmac-sha2-256-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-512, hmac-sha2-256
KexAlgorithms sntrup761x25519-sha512@openssh.com, curve25519-sha256@libssh.org, diffie-hellman-group-exchange-sha256, - sntrup4591761x25519-sha512@tinyssh.org, curve25519-sha256@libssh.org, diffie-hellman-group-exchange-sha256
TCPKeepAlive yes, - no
ClientAliveInterval 0, 300, 600, - 600
ClientAliveCountMax 3, - 3
AllowTcpForwarding yes, no, - no
AllowAgentForwarding yes, no, - no
GatewayPorts no, - no
X11Forwarding yes, no, - no
X11UseLocalhost yes, - yes
PermitTunnel no, - no
PermitUserEnvironment yes, no, - no
Compression delayed, yes, - no
UseDNS yes, - no
PrintMotd yes, - no
PrintLastLog yes, - no
Banner none, - none
DebianBanner x no
RevokedKeys - /etc/ssh/revoked_keys
Subsystem sftp /usr/libexec/openssh/sftp-server, /usr/lib/openssh/sftp-server, - internal-sftp -l INFO -f LOCAL6 -u 0027
Match Group sftponly - ForceCommand internal-sftp -l INFO -f LOCAL6 -u 0027, ChrootDirectory /home/%u, AllowTcpForwarding no, AllowAgentForwarding no, PasswordAuthentication no, PermitRootLogin no, X11Forwarding no

I did not find anything crazy in there, although a second pair of eyes glancing at the table would help. An important change though is that for apps.galaxyproject.eu, Port 8080 changes to Port 22. We should use a host vars file to revert this change via ssh_server_ports: ["8080"].

Comment on lines +9 to +13

- name: Ensure server key is gone
file:
path: /tmp/server_ca
state: absent
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Ensure server key is gone
file:
path: /tmp/server_ca
state: absent

all.yml Outdated
@@ -4,3 +4,5 @@
gather_facts: false
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
gather_facts: false
gather_facts: true

This would be needed to run dev-sec.ssh-hardening.

@kysrpex
Copy link
Copy Markdown
Contributor

kysrpex commented Jan 19, 2026

@mira-miracoli If you don't mind I'll merge the code review suggestions, I'd like to make some changes on top. We can re-discuss afterwards.

@kysrpex
Copy link
Copy Markdown
Contributor

kysrpex commented Jan 19, 2026

A second run of ansible-playbook all.yml --diff --check --ask-vault-password 2>&1 | tee all.log produces the following table (see this comment).

Table
Setting Old Value New Value
AddressFamily inet, any, - any
BatchMode no, - no
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc (default), chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr, - chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
CheckHostIP - yes
Compression yes, - no
ForwardAgent no, - no
ForwardX11 no, - no
GSSAPIAuthentication yes, - no
GSSAPIDelegateCredentials no, - no
HashKnownHosts yes, - -
HostbasedAuthentication no, - no
Include /etc/ssh/ssh_config.d/*.conf, - -
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256, - curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256, sntrup4591761x25519-sha512@tinyssh.org,curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256, sntrup761x25519-sha512@openssh.com,curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160, - hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
PasswordAuthentication yes, - no
PermitLocalCommand no, - no
Port 22, - 22
Protocol 2, - -
RSAAuthentication yes, - -
SendEnv LANG LC_*, - -
StrictHostKeyChecking ask, - ask
Tunnel no, - no
UseRoaming no, - -

I also have a copy of the playbook run all.log if that's easier than running the command.

@kysrpex
Copy link
Copy Markdown
Contributor

kysrpex commented Jan 19, 2026

TODO list:

  • use IPv4 only
  • Google Ciphers -> Use redhat crypto policies
  • CheckHostIP -> No
  • Compression -> Yes
  • Set Include to /etc/ssh/ssh_config.d/*.conf (keep it to use redhat's crypto policies). Use sshd_custom_options so it goes to the beginning.
  • Check if KexAlgorithms, MACs, Ciphers` are overridens.

@kysrpex
Copy link
Copy Markdown
Contributor

kysrpex commented Mar 9, 2026

@unode wrote (regarding #1939):

[...] check if the role denbi.ssh-no-root-with-password is not interfering with anything [...]

I am mentioning this detail because it will have to be kept into account for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants