[ssh_x509] OpenSSL 3.0 in FIPS mode

[ssh_x509 at roumenpetrov.info]

Hello,

FIPS support in PKIX-SSH is quite complex and so is not tested yet with OpenSSL 3.0 in FIPS mode.

Before to test I decided to adapt NSS engine to OpenSSL 3.0+ FIPS API.
Main reason is because NSS engine is simple as has FIPS related functionality - if OpenSSL runs in FIPS more engine required NSS database to be FIPS mode as well. With other words engine code only checks FIPS mode of OpenSSL and NSS libraries.

Off topic: it seems to me EC keys are broken in OpenSSL 3.0 branch, see issue #19138. Found by NSS tests. This affect PKIX-SSH as well for "store" based identities. So let ignore this part as is not FIPS related but impact programs build with OpenSSL 3+.

----
E_NSS release 4.1 adds experimental FIPS mode compatibility with OpenSSL 3+ and fix RSA signatures for OpenSSL FIPS 2* module.

Compatibility ensures detection of FIPS mode using new to OpenSSL 3+ API. Also it avoids crashes when "elementary" key cannot be extracted from PKEY.
Remark: to work with so called "external" keys code has to assign some specific "reference" data. Such data is associated to RSA, DSA and EC key.

Technically EVP_PKEY_get1_<key-type> functions return NULL when OpenSSL uses FIPS provider.
NSS engine was updated to raise error "invalid argument" in such case. As result error messages will look like:
Could not read private key from org.openssl.engine:e_nss:E_NSS Certificate (client-ec3) - E_NSS Test Team cyrillic-АБВ-Яабв-я greek-ΑΒΓ-Ωαβγ-ω
00D7947DCB7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (id-ecPublicKey : 109), Properties (<null>)
00D7947DCB7F0000:error:030C0100:digital envelope routines:evp_pkey_copy_downgraded:malloc failure:crypto/evp/p_lib.c:2052:
00D7947DCB7F0000:error:030000CD:digital envelope routines:evp_pkey_copy_downgraded:keymgmt export failure:crypto/evp/p_lib.c:2068:key type = id-ecPublicKey
00D7947DCB7F0000:error:40000070:E_NSS:nss_init_keyctx_ec:Invalid argument:.../engines/e_nss_ec.c:663:

Currently is not clear if missing functionality is by design(1) or is due to error in code(2).
I will ask OpenSSL for additional clarification.

Conclusion: currently  engine is not usable if OpenSSL 3 runs in FIPS mode.

----
About future:

If FIPS failure is due to (2) it could be enabled and I guess will be in next PKIX-SSH version. I cannot found any reason EC keys do not work nevertheless how key is constructed. My understanding is that FIPS is about allowed algorithms.

Perhaps to address (1) program code must be rewritten to use provider API instead "key method" API. Unfortunately this is quite expensive task.

Also note that PKIX-SSH supports RedHat and Solaris FIPS validation models, i.e. all those models must co-exist.


Regards,
Roumen Petrov