[Mod_nss-list] NSSSessionTickets causes some segfault in error log
Oliver Graute
oliver.graute at gmail.com
Fri Feb 26 13:19:12 UTC 2016
On 25/02/16, Rob Crittenden wrote:
> Oliver Graute wrote:
> > On 25/02/16, Oliver Graute wrote:
> >> On 24/02/16, Rob Crittenden wrote:
> >>> Oliver Graute wrote:
> >>>> On 23/02/16, Oliver Graute wrote:
> >>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute <oliver.graute at gmail.com> wrote:
> >>>>>> On 23/02/16, Rob Crittenden wrote:
> >>>>>>> Oliver Graute wrote:
> >>>>>>>> On 22/02/16, Rob Crittenden wrote:
> >>>>>>>>> Oliver Graute wrote:
> >>>>>>>>>> Hello,
> >>>>>>>>>>
> >>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver,
> >>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket
> >>>>>>>>>> feature in my nss.conf with:
> >>>>>>>>>>
> >>>>>>>>>> #RFC 5077
> >>>>>>>>>> NSSSessionTickets on
> >>>>>>>>>>
> >>>>>>>>>> then something is broken, I see segfaults in my apache error log:
> >>>>>>>>>>
> >>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations
> >>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38
> >>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5'
> >>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem)
> >>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled
> >>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy
> >>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11)
> >>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled
> >>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy
> >>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11)
> >>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled
> >>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy
> >>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> and in Chrome Browser I got:
> >>>>>>>>>>
> >>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH
> >>>>>>>>>>
> >>>>>>>>>> I tested also a basic ssl client connection with openssl:
> >>>>>>>>>>
> >>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug
> >>>>>>>>>>
> >>>>>>>>>> SSL_connect:SSLv3 read server certificate A
> >>>>>>>>>> SSL_connect:SSLv3 read server key exchange A
> >>>>>>>>>> SSL_connect:SSLv3 read server done A
> >>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B))
> >>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE
> >>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3.....
> >>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<...
> >>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\.....
> >>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe....
> >>>>>>>>>> SSL_connect:SSLv3 write client key exchange A
> >>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6))
> >>>>>>>>>> 0000 - 14 03 03 00 01 01 ......
> >>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A
> >>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D))
> >>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O..
> >>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+
> >>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........'
> >>>>>>>>>> SSL_connect:SSLv3 write finished A
> >>>>>>>>>> SSL_connect:SSLv3 flush data
> >>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0))
> >>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A
> >>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
> >>>>>>>>>>
> >>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment.
> >>>>>>>>>>
> >>>>>>>>>> some ideas, whats going on here?
> >>>>>>>>>
> >>>>>>>>> Can you get a stack trace from the core?
> >>>>>>>>
> >>>>>>>> I can give you an strace, see below. Other stack tools are currently not
> >>>>>>>> available, because I need to compile them first for my yocto
> >>>>>>>> environment. If you need something special please tell me.
> >>>>>>>>
> >>>>>>>>> This is Apache 2.4.x?
> >>>>>>>>
> >>>>>>>> yes it is Apache 2.4.16
> >>>>>>>>
> >>>>>>>>> Is it failing on a request or on startup?
> >>>>>>>>
> >>>>>>>> its failing on every https request.
> >>>>>>>
> >>>>>>> strace in this case isn't particularly helpful as it doesn't show where
> >>>>>>> it is crashing.
> >>>>>>>
> >>>>>>> Can I see your nss.conf?
> >>>>>>>
> >>>>>>> What version of NSS are you using?
> >>>>>>
> >>>>>> I'am using nss in version 3.19.2
> >>>>>>
> >>>>>> here my nss.conf
> >>>>>>
> >>>>>> #
> >>>>>> # This is the Apache server configuration file providing SSL support using.
> >>>>>> # the mod_nss plugin. It contains the configuration directives to instruct
> >>>>>> # the server how to serve pages over an https connection.
> >>>>>> #
> >>>>>> # Do NOT simply read the instructions in here without understanding
> >>>>>> # what they do. They're here only as hints or reminders. If you are unsure
> >>>>>> # consult the online docs. You have been warned.
> >>>>>> #
> >>>>>>
> >>>>>> #
> >>>>>> # When we also provide SSL we have to listen to the
> >>>>>> # standard HTTP port (see above) and to the HTTPS port
> >>>>>> #
> >>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
> >>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443"
> >>>>>> #
> >>>>>> Listen 443
> >>>>>>
> >>>>>> ##
> >>>>>> ## SSL Global Context
> >>>>>> ##
> >>>>>> ## All SSL configuration in this context applies both to
> >>>>>> ## the main server and all SSL-enabled virtual hosts.
> >>>>>> ##
> >>>>>>
> >>>>>> #
> >>>>>> # Some MIME-types for downloading Certificates and CRLs
> >>>>>> #
> >>>>>> AddType application/x-x509-ca-cert .crt
> >>>>>> AddType application/x-pkcs7-crl .crl
> >>>>>>
> >>>>>>
> >>>>>> # Pass Phrase Helper:
> >>>>>> # This helper program stores the token password pins between
> >>>>>> # restarts of Apache.
> >>>>>> # Unfortunately the directive is required even if we use no password
> >>>>>> # (though in such case the program should never be used)
> >>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache
> >>>>>>
> >>>>>> # Pass Phrase Dialog:
> >>>>>> # Configure the pass phrase gathering process.
> >>>>>> # The filtering dialog program (`builtin' is a internal
> >>>>>> # terminal dialog) has to provide the pass phrase on stdout.
> >>>>>> #NSSPassPhraseDialog builtin
> >>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf
> >>>>>>
> >>>>>> # Configure the SSL Session Cache.
> >>>>>> # NSSSessionCacheSize is the number of entries in the cache.
> >>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds).
> >>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds).
> >>>>>> NSSSessionCacheSize 10000
> >>>>>> NSSSessionCacheTimeout 100
> >>>>>> NSSSession3CacheTimeout 86400
> >>>>>>
> >>>>>> #RFC 5077
> >>>>>> NSSSessionTickets off
> >>>>>
> >>>>> SessionTickets are on not off
> >>>>>
> >>>>> NSSSessionTickets on
> >>>>>
> >>>>
> >>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss
> >>>> parameters for every virtual host. It seems that the problem is here. Because
> >>>> if I remove the include of httpd-vhosts.conf no segfaults occur.
> >>>>
> >>>> here my httpd-vhosts.conf with nss related parameters. Are these
> >>>> parameters capable for a virtual host configuration?
> >>>
> >>> You've included some directives that are expected to be global:
> >>>
> >>> NSSPassPhraseHelper
> >>> NSSPassPhraseDialog
> >>> NSSSession*Cache*
> >>> NSSCertificateDatabase
> >>>
> >>> I doubt they hurt anything.
> >>
> >> ok thx, I removed these directives from the httpd-vhosts.conf
> >>
> >>>
> >>> I still can't duplicate a crash. I ran it through valgrind to see if
> >>> there was a memory issue and came up with nothing too.
> >>>
> >>> This could be an architecture problem that I just can't see on x86
> >>> hardware I suppose.
> >>
> >> I'am using a arm hardware (imx6)
> >>
> >>> This particular crash seems rather strange too. Within mod_nss enabling
> >>> this just calls:
> >>>
> >>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS,
> >>> mctx->sc->session_tickets);
> >>>
> >>> Basically enabling it in the model server socket. I would imagine the
> >>> crash is probably deeper in NSS.
> >>
> >>
> >>> To narrow things down you might try the NSS tool selfserv. It has an
> >>> option, -u, to enable session tickets. It might eliminate mod_nss as the
> >>> crash source (or it might implicate it too).
> >>
> >>
> >> thx for the hint to the NSS selfserv tool.
> >>
> >> I'am using it with DSA Keys instead of Eliptic Curve Keys because
> >> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers)
> >>
> >> First I generate a new nss db.
> >>
> >> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
> >> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
> >> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
> >> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
> >> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf
> >>
> >> Then starting selfserv with params:
> >>
> >> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u
> >>
> >> selfserv: About to call accept.
> >>
> >> Now I perform a https request with the Chrome Browser
> >>
> >> GET /test.php HTTP/1.1
> >> Host: 192.168.1.229
> >> Connection: keep-alive
> >> Cache-Control: max-age=0
> >> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> >> Upgrade-Insecure-Requests: 1
> >> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
> >> DNT: 1
> >> Accept-Encoding: gzip, deflate, sdch
> >> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
> >>
> >> EOF
> >>
> >> it looks for me that it works with selfserv. But I'm not sure if the Session
> >> Ticket works and if it makes a difference not using ECC Keys.
> >
> > I checked with wireshark, and I saw the TLS Session Ticket Extension
> > in the hello packet there.
> >
> > I'also switched back to my old ciphers in my apache setup nss.conf
> >
> > NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha
> >
> > and this is working together with NSSSessionTickets on.
> >
> > So it breaks if i'am using ECC Ciphersuite together with Session Tickets!
> >
> > NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256
> > NSSSessionTickets on
> >
> > is this something that can be explained?
>
> Hmm, interesting. It gives me something to look at. I was in the middle
> of responding to your previous message when this one came in.
>
> I had been testing with an RSA key. Let me try an ECC key to see if I
> can reproduce the crash (no success yet).
>
> Here is the previous bit I was going to send:
>
> This actually generated an RSA key for the cert you used for selfserv.
> certutil will generate a certificate if the -k <id> option isn't provided.
>
> You can do the same with ECC by adding to the last command -k ec -q <curve>
I updated from nss 3.19 to nss 3.21 but I run into the same issue with it.
Then I played around with gdb and saw that it breaks in libssl3.so
Some hints how I can debug this further?
Best Regards,
Oliver
gdb httpd
run -X -e debug -D /usr/sbin/httpd
root at ibis-box:/var/apache2/logs# gdb httpd
GNU gdb (GDB) 7.9.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-poky-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from httpd...(no debugging symbols found)...done.
(gdb) run -X -e debug -D /usr/sbin/httpd
Starting program: /usr/sbin/httpd -X -e debug -D /usr/sbin/httpd
warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /lib/libthread_db-1.0.so
line to your configuration file "/home/root/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
[Fri Feb 26 00:03:18.648483 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so
[Fri Feb 26 00:03:18.700565 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so
[Fri Feb 26 00:03:18.750258 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so
[Fri Feb 26 00:03:18.801825 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so
[Fri Feb 26 00:03:18.852520 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so
[Fri Feb 26 00:03:18.909582 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so
[Fri Feb 26 00:03:18.964388 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so
[Fri Feb 26 00:03:19.022456 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so
[Fri Feb 26 00:03:19.081787 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so
[Fri Feb 26 00:03:19.142737 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so
[Fri Feb 26 00:03:19.205120 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so
[Fri Feb 26 00:03:19.288557 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so
[Fri Feb 26 00:03:19.355114 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so
[Fri Feb 26 00:03:19.428930 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module log_config_module from /usr/lib/apache2/modules/mod_log_config.so
[Fri Feb 26 00:03:19.495578 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so
[Fri Feb 26 00:03:19.569779 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so
[Fri Feb 26 00:03:19.641764 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so
[Fri Feb 26 00:03:19.716852 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module version_module from /usr/lib/apache2/modules/mod_version.so
[Fri Feb 26 00:03:20.213946 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module nss_module from /usr/lib/apache2/modules/libmodnss.so
[Fri Feb 26 00:03:20.307607 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so
[Fri Feb 26 00:03:20.393425 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module unixd_module from /usr/lib/apache2/modules/mod_unixd.so
[Fri Feb 26 00:03:20.483090 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so
[Fri Feb 26 00:03:20.578632 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so
[Fri Feb 26 00:03:20.671102 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so
[Fri Feb 26 00:03:20.763080 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so
Program received signal SIGSEGV, Segmentation fault.
0x76b366dc in ?? () from /usr/lib/libssl3.so
More information about the Mod_nss-list
mailing list