[ssh_x509] PKIX-SSH, OpenSSL 3 API, and FIPS mode

[ssh_x509 at roumenpetrov.info]

Hello,


A) transition
Transition of PKIX-SSH code to use OpenSSL 3* API is complete. All modifications are published in main branch.
Impact on another branches is managed as well. Please note that some still use archaic OpenSSL 0.9 API - so this code is out of scope.
As promised earlier, transition does not include DSA and DH(Diffie-Helman).

In consequence could be used OpenSSL FIPS provider.

B) activation
As before, if environment variable OPENSSL_FIPS is set FIPS mode is activated. This works well with FIPS provider. Case with default provider needs additional investigation as now programs runs without error.
Note that all OpenSSL 3* builds are considered FIPS enabled while certification is for OpenSSL 3.0.8/9 (FIPS-2) and OpenSSL 3.1.3 (FISP-3).

In addition FIPS mode could be activated through OpenSSL 3* configuration.

C) issues
a) dsa
Use of DSA in FIPS mode. Program code exclude DSA when cryptographic library runs in FIPS mode.
The issue is that FIPS provider does not allows creation of SSH DSA keys even if non-FIPS mode.
This is visible by command:
$ openssl genpkey -genparam -algorithm DSA -pkeyopt dsa_paramgen_bits:1024 -pkeyopt dsa_paramgen_q_bits:160

In consequence SSH DSA algorithms are not enabled when program code detect loaded fips provider.


b) FIPS flags
OpenSSL 3* still define FIPS flag for digests and ciphers.
These flags has to effect even if FIPS provider is run in fips mode.
It seems to me this is a regression.
Unfortunately OpenSSL FIPS providers are validated and defect cannot be resolved.

c) unit-tests and RSA key size
This is not new FIPS requirement - the minimum RSA key size is 2048.
It seem to me FIPS provider does not accept RSA keys with size less than 2048 even in non-FIPS mode.
Some unit-tests fail even with increase of key size for pre-generated RSA keys.
This part is under investigation.
Note updated RSA keys are not published(pushed) yet.

d) 3des-cbc
According OpenSSL documentation DES-EDE3-CBC should be available if FIPS mode.
Unfortunately fips-try-ciphers tests fail when session try to used 3des-cbc ssh cipher.
This part is planed for investigation.
Remark: Cipher 3des-cbc works just fine with OpenSSL 1.0.2 FIPS validated module.

e) tests in FIPS environment
Modern enterprise Linux-es may not allow dynamic switch in FIPS or non-FIPS mode. Mode should be activated on installation time and cannot be deactivated.
PKIX-SSH tests are not ready for this. Also note, that tests are note ready even in case when FIPS mode is activated trough configuration.

Please use non-FIPS environment, with activated FIPS provider in non-FIPS mode.


So, the mainstream code is OpenSSL 3* ready. This does not means that code uses only OpenSSL 3* API!
The program code still support engines and this code requires pre-OpenSSL 3* API for backward compatibility.


Unfortunately, issues with RSA key type and cipher prevent me from announcing FIPS support.


Regards,
Roumen Petrov