[Mod_nss-list] NSSSessionTickets causes some segfault in error log
Rob Crittenden
rcritten at redhat.com
Fri Feb 26 19:27:40 UTC 2016
Oliver Graute wrote:
> On 25/02/16, Rob Crittenden wrote:
>> Oliver Graute wrote:
>>> On 25/02/16, Oliver Graute wrote:
>>>> On 24/02/16, Rob Crittenden wrote:
>>>>> Oliver Graute wrote:
>>>>>> On 23/02/16, Oliver Graute wrote:
>>>>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute <oliver.graute at gmail.com> wrote:
>>>>>>>> On 23/02/16, Rob Crittenden wrote:
>>>>>>>>> Oliver Graute wrote:
>>>>>>>>>> On 22/02/16, Rob Crittenden wrote:
>>>>>>>>>>> Oliver Graute wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver,
>>>>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket
>>>>>>>>>>>> feature in my nss.conf with:
>>>>>>>>>>>>
>>>>>>>>>>>> #RFC 5077
>>>>>>>>>>>> NSSSessionTickets on
>>>>>>>>>>>>
>>>>>>>>>>>> then something is broken, I see segfaults in my apache error log:
>>>>>>>>>>>>
>>>>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations
>>>>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38
>>>>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5'
>>>>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem)
>>>>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled
>>>>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy
>>>>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11)
>>>>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled
>>>>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy
>>>>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11)
>>>>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled
>>>>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy
>>>>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and in Chrome Browser I got:
>>>>>>>>>>>>
>>>>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH
>>>>>>>>>>>>
>>>>>>>>>>>> I tested also a basic ssl client connection with openssl:
>>>>>>>>>>>>
>>>>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug
>>>>>>>>>>>>
>>>>>>>>>>>> SSL_connect:SSLv3 read server certificate A
>>>>>>>>>>>> SSL_connect:SSLv3 read server key exchange A
>>>>>>>>>>>> SSL_connect:SSLv3 read server done A
>>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B))
>>>>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE
>>>>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3.....
>>>>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<...
>>>>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\.....
>>>>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe....
>>>>>>>>>>>> SSL_connect:SSLv3 write client key exchange A
>>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6))
>>>>>>>>>>>> 0000 - 14 03 03 00 01 01 ......
>>>>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A
>>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D))
>>>>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O..
>>>>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+
>>>>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........'
>>>>>>>>>>>> SSL_connect:SSLv3 write finished A
>>>>>>>>>>>> SSL_connect:SSLv3 flush data
>>>>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0))
>>>>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A
>>>>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
>>>>>>>>>>>>
>>>>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment.
>>>>>>>>>>>>
>>>>>>>>>>>> some ideas, whats going on here?
>>>>>>>>>>>
>>>>>>>>>>> Can you get a stack trace from the core?
>>>>>>>>>>
>>>>>>>>>> I can give you an strace, see below. Other stack tools are currently not
>>>>>>>>>> available, because I need to compile them first for my yocto
>>>>>>>>>> environment. If you need something special please tell me.
>>>>>>>>>>
>>>>>>>>>>> This is Apache 2.4.x?
>>>>>>>>>>
>>>>>>>>>> yes it is Apache 2.4.16
>>>>>>>>>>
>>>>>>>>>>> Is it failing on a request or on startup?
>>>>>>>>>>
>>>>>>>>>> its failing on every https request.
>>>>>>>>>
>>>>>>>>> strace in this case isn't particularly helpful as it doesn't show where
>>>>>>>>> it is crashing.
>>>>>>>>>
>>>>>>>>> Can I see your nss.conf?
>>>>>>>>>
>>>>>>>>> What version of NSS are you using?
>>>>>>>>
>>>>>>>> I'am using nss in version 3.19.2
>>>>>>>>
>>>>>>>> here my nss.conf
>>>>>>>>
>>>>>>>> #
>>>>>>>> # This is the Apache server configuration file providing SSL support using.
>>>>>>>> # the mod_nss plugin. It contains the configuration directives to instruct
>>>>>>>> # the server how to serve pages over an https connection.
>>>>>>>> #
>>>>>>>> # Do NOT simply read the instructions in here without understanding
>>>>>>>> # what they do. They're here only as hints or reminders. If you are unsure
>>>>>>>> # consult the online docs. You have been warned.
>>>>>>>> #
>>>>>>>>
>>>>>>>> #
>>>>>>>> # When we also provide SSL we have to listen to the
>>>>>>>> # standard HTTP port (see above) and to the HTTPS port
>>>>>>>> #
>>>>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
>>>>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443"
>>>>>>>> #
>>>>>>>> Listen 443
>>>>>>>>
>>>>>>>> ##
>>>>>>>> ## SSL Global Context
>>>>>>>> ##
>>>>>>>> ## All SSL configuration in this context applies both to
>>>>>>>> ## the main server and all SSL-enabled virtual hosts.
>>>>>>>> ##
>>>>>>>>
>>>>>>>> #
>>>>>>>> # Some MIME-types for downloading Certificates and CRLs
>>>>>>>> #
>>>>>>>> AddType application/x-x509-ca-cert .crt
>>>>>>>> AddType application/x-pkcs7-crl .crl
>>>>>>>>
>>>>>>>>
>>>>>>>> # Pass Phrase Helper:
>>>>>>>> # This helper program stores the token password pins between
>>>>>>>> # restarts of Apache.
>>>>>>>> # Unfortunately the directive is required even if we use no password
>>>>>>>> # (though in such case the program should never be used)
>>>>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache
>>>>>>>>
>>>>>>>> # Pass Phrase Dialog:
>>>>>>>> # Configure the pass phrase gathering process.
>>>>>>>> # The filtering dialog program (`builtin' is a internal
>>>>>>>> # terminal dialog) has to provide the pass phrase on stdout.
>>>>>>>> #NSSPassPhraseDialog builtin
>>>>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf
>>>>>>>>
>>>>>>>> # Configure the SSL Session Cache.
>>>>>>>> # NSSSessionCacheSize is the number of entries in the cache.
>>>>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds).
>>>>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds).
>>>>>>>> NSSSessionCacheSize 10000
>>>>>>>> NSSSessionCacheTimeout 100
>>>>>>>> NSSSession3CacheTimeout 86400
>>>>>>>>
>>>>>>>> #RFC 5077
>>>>>>>> NSSSessionTickets off
>>>>>>>
>>>>>>> SessionTickets are on not off
>>>>>>>
>>>>>>> NSSSessionTickets on
>>>>>>>
>>>>>>
>>>>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss
>>>>>> parameters for every virtual host. It seems that the problem is here. Because
>>>>>> if I remove the include of httpd-vhosts.conf no segfaults occur.
>>>>>>
>>>>>> here my httpd-vhosts.conf with nss related parameters. Are these
>>>>>> parameters capable for a virtual host configuration?
>>>>>
>>>>> You've included some directives that are expected to be global:
>>>>>
>>>>> NSSPassPhraseHelper
>>>>> NSSPassPhraseDialog
>>>>> NSSSession*Cache*
>>>>> NSSCertificateDatabase
>>>>>
>>>>> I doubt they hurt anything.
>>>>
>>>> ok thx, I removed these directives from the httpd-vhosts.conf
>>>>
>>>>>
>>>>> I still can't duplicate a crash. I ran it through valgrind to see if
>>>>> there was a memory issue and came up with nothing too.
>>>>>
>>>>> This could be an architecture problem that I just can't see on x86
>>>>> hardware I suppose.
>>>>
>>>> I'am using a arm hardware (imx6)
>>>>
>>>>> This particular crash seems rather strange too. Within mod_nss enabling
>>>>> this just calls:
>>>>>
>>>>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS,
>>>>> mctx->sc->session_tickets);
>>>>>
>>>>> Basically enabling it in the model server socket. I would imagine the
>>>>> crash is probably deeper in NSS.
>>>>
>>>>
>>>>> To narrow things down you might try the NSS tool selfserv. It has an
>>>>> option, -u, to enable session tickets. It might eliminate mod_nss as the
>>>>> crash source (or it might implicate it too).
>>>>
>>>>
>>>> thx for the hint to the NSS selfserv tool.
>>>>
>>>> I'am using it with DSA Keys instead of Eliptic Curve Keys because
>>>> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers)
>>>>
>>>> First I generate a new nss db.
>>>>
>>>> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
>>>> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
>>>> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
>>>> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
>>>> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf
>>>>
>>>> Then starting selfserv with params:
>>>>
>>>> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u
>>>>
>>>> selfserv: About to call accept.
>>>>
>>>> Now I perform a https request with the Chrome Browser
>>>>
>>>> GET /test.php HTTP/1.1
>>>> Host: 192.168.1.229
>>>> Connection: keep-alive
>>>> Cache-Control: max-age=0
>>>> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
>>>> Upgrade-Insecure-Requests: 1
>>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
>>>> DNT: 1
>>>> Accept-Encoding: gzip, deflate, sdch
>>>> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
>>>>
>>>> EOF
>>>>
>>>> it looks for me that it works with selfserv. But I'm not sure if the Session
>>>> Ticket works and if it makes a difference not using ECC Keys.
>>>
>>> I checked with wireshark, and I saw the TLS Session Ticket Extension
>>> in the hello packet there.
>>>
>>> I'also switched back to my old ciphers in my apache setup nss.conf
>>>
>>> NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha
>>>
>>> and this is working together with NSSSessionTickets on.
>>>
>>> So it breaks if i'am using ECC Ciphersuite together with Session Tickets!
>>>
>>> NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256
>>> NSSSessionTickets on
>>>
>>> is this something that can be explained?
>>
>> Hmm, interesting. It gives me something to look at. I was in the middle
>> of responding to your previous message when this one came in.
>>
>> I had been testing with an RSA key. Let me try an ECC key to see if I
>> can reproduce the crash (no success yet).
>>
>> Here is the previous bit I was going to send:
>>
>> This actually generated an RSA key for the cert you used for selfserv.
>> certutil will generate a certificate if the -k <id> option isn't provided.
>>
>> You can do the same with ECC by adding to the last command -k ec -q <curve>
>
>
> I updated from nss 3.19 to nss 3.21 but I run into the same issue with it.
>
> Then I played around with gdb and saw that it breaks in libssl3.so
> Some hints how I can debug this further?
>
> Best Regards,
>
> Oliver
>
> gdb httpd
> run -X -e debug -D /usr/sbin/httpd
>
> root at ibis-box:/var/apache2/logs# gdb httpd
> GNU gdb (GDB) 7.9.1
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "arm-poky-linux-gnueabi".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from httpd...(no debugging symbols found)...done.
> (gdb) run -X -e debug -D /usr/sbin/httpd
> Starting program: /usr/sbin/httpd -X -e debug -D /usr/sbin/httpd
> warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
> To enable execution of this file add
> add-auto-load-safe-path /lib/libthread_db-1.0.so
> line to your configuration file "/home/root/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/home/root/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> [Fri Feb 26 00:03:18.648483 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so
> [Fri Feb 26 00:03:18.700565 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so
> [Fri Feb 26 00:03:18.750258 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so
> [Fri Feb 26 00:03:18.801825 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so
> [Fri Feb 26 00:03:18.852520 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so
> [Fri Feb 26 00:03:18.909582 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so
> [Fri Feb 26 00:03:18.964388 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so
> [Fri Feb 26 00:03:19.022456 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so
> [Fri Feb 26 00:03:19.081787 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so
> [Fri Feb 26 00:03:19.142737 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so
> [Fri Feb 26 00:03:19.205120 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so
> [Fri Feb 26 00:03:19.288557 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so
> [Fri Feb 26 00:03:19.355114 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so
> [Fri Feb 26 00:03:19.428930 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module log_config_module from /usr/lib/apache2/modules/mod_log_config.so
> [Fri Feb 26 00:03:19.495578 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so
> [Fri Feb 26 00:03:19.569779 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so
> [Fri Feb 26 00:03:19.641764 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so
> [Fri Feb 26 00:03:19.716852 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module version_module from /usr/lib/apache2/modules/mod_version.so
> [Fri Feb 26 00:03:20.213946 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module nss_module from /usr/lib/apache2/modules/libmodnss.so
> [Fri Feb 26 00:03:20.307607 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so
> [Fri Feb 26 00:03:20.393425 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module unixd_module from /usr/lib/apache2/modules/mod_unixd.so
> [Fri Feb 26 00:03:20.483090 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so
> [Fri Feb 26 00:03:20.578632 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so
> [Fri Feb 26 00:03:20.671102 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so
> [Fri Feb 26 00:03:20.763080 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x76b366dc in ?? () from /usr/lib/libssl3.so
Can I ask why you have NSSEnforceValidCerts off set? Is it because of an
error that was logged, preventing the server from starting?
rob
More information about the Mod_nss-list
mailing list