[ssh_x509] ssh handshake failed in FIPS mode

ssh_x509 at roumenpetrov.info ssh_x509 at roumenpetrov.info
Mon May 6 13:39:43 EEST 2019

Hi gas,

ssh_x509 at roumenpetrov.info wrote:
> Hi
> We are using openssh and applied pkixssh patch. ( Openssh version is
> 7.5p1 and applied openssh-7.5p1+x509-10.1.1.diff   ). Our Openssl was
> based on 1.0.2n . We were using ssh for creating tunnels between
> servers and every thing were working fine ( in FIPS as well as NonFIPS
> mode ).
> Recently we have upgraded openssl to 1.0.2q  and handshake started
> failing. I have not clue on why it fails.  Following are the
> parameters set in the configuration files and output from debug log
> are also provided below.
> <<<<
> In server side
> [SNIP]
> debug3: mm_answer_sign
> debug3: ssh_x509_sign: key alg/type/name:
> x509v3-sign-rsa/RSA+cert/x509v3-sign-rsa
> debug3: ssh_x509_sign: compatibility: { 0x00000000, 0x00000000 }
> debug3: ssh_x509_sign: alg=x509v3-sign-rsa, md=rsa-sha1
> ssh_x509_EVP_PKEY_sign: EVP_SignInit_ex fail with
> errormsg='error:060B5098:lib(6):func(181):reason(152)'
../1.0.2q/bin/openssl errstr '060B5098:lib(6):func(181):reason(152)'
error:060B5098:digital envelope routines:func(181):invalid digest

So function with code 181 does not exist in official 1.0.2.q release.

With current stable:
.../1.0.2_stable/bin/openssl errstr '060B5098:lib(6):func(181):reason(152)'
error:060B5098:digital envelope routines:EVP_DecryptUpdate:invalid digest

May be openssl version mismatch. Sometime crypto-library could be loaded 
twice - once directly from ssh binaries and indirectly from ldap or dns 

> debug3: ssh_x509_sign: return -22
> mm_answer_sign: Xkey_sign failed: error in libcrypto
> debug1: do_cleanup
> debug1: Killing privsep child 28609
> <<<<
> What could be the problem. Any pointers on this would be of great help for me.

It is difficult to find out what could be reason. May be just 
compilation issue.

Package has some FIPS regression tests. They could be start manually.

OPENSSL=.../1.0.2_stable_fips/bin/openssl \
SSH_X509TESTS=skip \
make -e FIPS_LTESTS=fips-connect-privsep REGRESS_TARGETS=f-exec tests

Another FIPS related test is fips-try-ciphers. Result of this rest is 
more important especially line for :
test try ciphers in FIPS mode: fips-cipher aes256-ctr fips-mac hmac-sha2-512

Does FIPS regression tests pass?

> Thanks and regards
> gas

Roumen Petrov

More information about the ssh_x509 mailing list