FIPS mode self check error

[MattTheCuber]

We use a Red Hat Enterprise Linux 8.5 machine with FIPS mode enabled. After installing any program (folder or one-file mode) and running it, we get the following error:

Error in GnuTLS initialization: Error while performing self checks.

Note, this is a non-critical bug as it does not seem to affect the usage of the program except the undesired terminal output. Googling the error will return a few posts, but they don't seem to have any solutions.

gnutls-cli version: 3.6.16

[rokm]

Hmm, can you give me a minimal reproducer that uses GnuTLS from python? Are you building under RHEL 8.5 as well, or under some other distribution? Also, what PyInstaller version are you using - gnutls seems to be using .hmac files, and we added collection of those in 6.4.0.

[MattTheCuber]

Great point, I kept seeing the error with everything I tried to compile, but I never tried to create a minimal example. I am using the latest pyinstaller version (6.9.0) and building and running on the same RHEL (8.5) system. Here were my findings in this process:

Issue 1

test.py:

import IPython

import comm

import ipykernel

import jupyter_client

import matplotlib

import matplotlib_inline

Compiling and running one of these returns the GnuTLS error originally stated.

$ pyinstaller test.py
...
$ ./dist/test/test
Error in GnuTLS initialization: Error while performing self checks.

Issue 2

test.py:

import hashlib

import OpenSSL

import parso

Compiling and running one of these returns:

$ pyinstaller test.py
...
$ ./dist/test/test
crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Aborted (core dumped)

What is strange is that it doesn't happen when thrid-party libraries import hashlib, openssl, or parso.

[rokm]

Can you check if .libname.hmac files are collected along with the shared libraries? I.e., assuming that libgnutls.so.XY is collected into frozen application, there should be an accompanying .libgnutls.so.XY.hmac file (note the starting dot) in the same directory. Same for the openssl libs.

[MattTheCuber]

For the import matplotlib program, the following files are collected:

dist/test/_internal/.libcrypto.so.1.1.hmac
dist/test/_internal/.libgcrypt.so.20.hmac
dist/test/_internal/.libgnutls.so.30.hmac
dist/test/_internal/.libhogweed.so.4.hmac
dist/test/_internal/.libnettle.so.6.hmac
dist/test/_internal/.libssl.so.1.1.hmac

For the import hashlib program, only the .libcrypto.so.1.1.hmac file is collected.

[rokm]

Hmm, last time time I was looking into FIPS mode (pyinstaller/pyinstaller#8273) collecting these seemed to suffice.

I'll try to set up a test RHEL8 system again.

[MattTheCuber]

If it is easier for you to explain debugging steps, I would be happy to help you test.

[rokm]

If it is easier for you to explain debugging steps, I would be happy to help you test.

I think it is, because I have no idea what to look for at the moment.

[rokm]

Can you tell me what python version you used, and how did you install it? (If I recall correctly, default is 3.6 which does not work with PyInstaller 6.9). And tested packages are installed from PyPI via pip?

[MattTheCuber]

$ python --version
Python 3.11.6

installed using the yum package manager

Packages are installed from PyPi using pip, yes.

[rokm]

Can you give me pip freeze list for the environment? The comm and matplotlib examples are not pulling in GnuTLS on my test system, so it might be pulled in via some optional dependency.

That said, hashlib example does pull in openssl libs and their hmac files, but it runs on my test system (admittedly, it is 8.10 with python 3.11.9, with FIPS enabled post-hoc via "sudo fips-mode-setup --enable && sudo reboot"). In my case, both libssl.so.1.1 and libcrypto.so.1.1 and their corresponding hmac files are collected. And if I remove either hmac file, I get

crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Aborted (core dumped)

[rokm]

Hmm, let's focus on hashlib and OpenSSL first. If your hashlib build did not collect libssl.so.1.1 and its hmac file (I see that _hashlib extension is, indeed, linked only against libcrypto.so.1.1), can you try to either:

• use --add-binary /usr/lib64/libssl.so.1.1:. to ensure that it is collected
• remove the bundled libcrypto.so.1.1 and try running the application

Could be that issue is that one lib is bundled and the other is not.

[rokm]

Could be that issue is that one lib is bundled and the other is not.

Yep, that seems to be the case on my system; the libssl.so.1.1.so is pulled in by python's _blake2 extension, so if I add --exclude _blake2 to my PyInstaller command for hashlib example, only libcrypto.so.1.1 is collected, and I get the selftest failure.

[MattTheCuber]

Can you give me pip freeze list for the environment?

I am in a giant catch-all environment (probably should have made a fresh one). Let me know if you want me to make a new one or send you a paste bin link of my current environment.

admittedly, it is 8.10 with python 3.11.9, with FIPS enabled post-hoc via "sudo fips-mode-setup --enable && sudo reboot"

Not an expert, but enabling FIPS post-boot shouldn't be problem, I doubt the Python version change makes a difference, but we have noticed big differences between libraries on different RHEL 8.x versions.

Trying you suggestions for hashlib now...

[MattTheCuber]

Adding --add-binary /usr/lib64/libssl.so.1.1:. worked for import hashlib (no errors)

[MattTheCuber]

remove the bundled libcrypto.so.1.1 and try running the application

This also worked