YubiHSM 2: signing and key generation fail

[BASOPSESSION]

YubiHSM 2: signing and key generation fail — object search before login + SO login on a token without SO support

Hi,

I hit the same problem as this issue 531 with a YubiHSM 2 and dug into it.

Environment

XCA 2.9.0 (Windows 11)
YubiHSM 2, connector 3.0.7
yubihsm_pkcs11 module tested in versions 2.4.2, 2.6.0 and 2.7.3 — identical behavior on all three
Key on token: RSA4096, id 0x0002, all 16 domains, capabilities sign-pkcs,sign-pss,...

Authentication works fine: in the "Manage security token" dialog, XCA prompts for the PIN, logs in (CKU_USER) and lists/imports the keys correctly.

Symptom when signing a certificate

XCA loops on "Please insert card: Yubico (www.yubico.com) YubiHSM [YubiHSM] with Serial: xxxx" and never asks for the PIN.

PKCS#11 module debug log during the signing attempt (module 2.7.3, same with 2.4.2/2.6.0):

C_OpenSession: Allocated session 1
get_session: Session 1 user not logged in
C_FindObjects: Returning 0 object(s)
C_LoginUser: Invalid user type, only regular user allowed
C_CloseSession: Closing session 1

For comparison, the working "Manage security token" flow in the very same run logs a successful CKU_USER login followed by yh_util_list_objects: Found 2 objects and full attribute reads. So the token, module, connector and PIN are all fine — only the signing path fails.

Root cause 1 — pki_scard::find_key_on_card() searches without login

lib/pki_scard.cpp: find_key_on_card() (called by prepare_card() before any signing operation) opens a session and searches for the CKO_PUBLIC_KEY object without logging in:

foreach(sl, p11.getSlotList()) {
pkcs11 p11sess;
p11sess.startSession(sl);
foreach(CK_OBJECT_HANDLE object, p11sess.objectList(cls)) { ...

This works on classic smartcards where public objects are visible without authentication. The YubiHSM 2, however, hides all objects (public keys included) until a CKU_USER login is performed — the device requires an authenticated session for everything. So objectList() always returns 0 objects, prepare_card() concludes the card is not inserted, and the PIN dialog is never reached. Note that the subsequent code path (pki_scard::decryptKey()) does the right thing (CKU_USER login, then private key lookup) — it is just never reached.

Root cause 2 — SO login forced on a token that has no SO

lib/pkcs11.h:

bool need_SO_for_object_mod() const
{
// Yubikey Need SO Pin to modify objects
return manufacturerID() == "Yubico (www.yubico.com)";
}

This workaround targets the YubiKey (PIV/ykcs11), but the YubiHSM 2 reports the same manufacturerID while not supporting the SO role at all (confirmed by Yubico support earlier in this issue). As a result every tokenLoginForModification() call sends C_Login(CKU_SO), which the module rejects:

C_LoginUser: Invalid user type, only regular user allowed

This breaks, on a YubiHSM 2:

Key generation from XCA (pki_scard::generate()): after startSession(), getRandom() already fails (C_GenerateRandom: Unknown session — not logged in, harmless), select_id()/findUniqueID(CKO_PUBLIC_KEY) enumerates existing objects without login so it cannot see existing IDs, and then tokenLoginForModification() aborts with the SO login rejection. Key generation is therefore impossible from XCA.
Storing the freshly signed certificate on the token (pki_x509::store_token()), renaming (renameOnToken(), storeAttribute) and deleting objects — all gated by tokenLoginForModification().

The model string can be used to discriminate: the YubiHSM 2 reports model "YubiHSM" while YubiKeys report "YubiKey ...".

Proposed fix but i'm not a good developper...

yubihsm_p11.log

In find_key_on_card(): if the public key search returns nothing on a token that sets CKF_LOGIN_REQUIRED and whose serial matches the stored card_serial, perform a regular user login (tokenLogin(card_label, false)) and retry the search. Restricting to the matching serial avoids PIN prompts on unrelated tokens.
Add tkInfo::loginRequired() returning token_info.flags & CKF_LOGIN_REQUIRED.
In need_SO_for_object_mod(): keep the YubiKey workaround but exclude the YubiHSM:
return manufacturerID() == "Yubico (www.yubico.com)" && model() != "YubiHSM";
For key generation, the same login-before-enumeration principle applies to select_id() / findUniqueID(): on a CKF_LOGIN_REQUIRED token a CKU_USER login should happen before enumerating objects, otherwise ID collision detection cannot work.

Happy to provide full debug logs atatched.
Same issue when trying to generate a private key directly on the HSM.

Regards

Benoit