[Mod_nss-list] NSSSessionTickets causes some segfault in error log
Rob Crittenden
rcritten at redhat.com
Thu Feb 25 15:33:32 UTC 2016
Oliver Graute wrote:
> On 25/02/16, Oliver Graute wrote:
>> On 24/02/16, Rob Crittenden wrote:
>>> Oliver Graute wrote:
>>>> On 23/02/16, Oliver Graute wrote:
>>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute <oliver.graute at gmail.com> wrote:
>>>>>> On 23/02/16, Rob Crittenden wrote:
>>>>>>> Oliver Graute wrote:
>>>>>>>> On 22/02/16, Rob Crittenden wrote:
>>>>>>>>> Oliver Graute wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver,
>>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket
>>>>>>>>>> feature in my nss.conf with:
>>>>>>>>>>
>>>>>>>>>> #RFC 5077
>>>>>>>>>> NSSSessionTickets on
>>>>>>>>>>
>>>>>>>>>> then something is broken, I see segfaults in my apache error log:
>>>>>>>>>>
>>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations
>>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38
>>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5'
>>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem)
>>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled
>>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy
>>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11)
>>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled
>>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy
>>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11)
>>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled
>>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy
>>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> and in Chrome Browser I got:
>>>>>>>>>>
>>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH
>>>>>>>>>>
>>>>>>>>>> I tested also a basic ssl client connection with openssl:
>>>>>>>>>>
>>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug
>>>>>>>>>>
>>>>>>>>>> SSL_connect:SSLv3 read server certificate A
>>>>>>>>>> SSL_connect:SSLv3 read server key exchange A
>>>>>>>>>> SSL_connect:SSLv3 read server done A
>>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B))
>>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE
>>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3.....
>>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<...
>>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\.....
>>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe....
>>>>>>>>>> SSL_connect:SSLv3 write client key exchange A
>>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6))
>>>>>>>>>> 0000 - 14 03 03 00 01 01 ......
>>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A
>>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D))
>>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O..
>>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+
>>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........'
>>>>>>>>>> SSL_connect:SSLv3 write finished A
>>>>>>>>>> SSL_connect:SSLv3 flush data
>>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0))
>>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A
>>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
>>>>>>>>>>
>>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment.
>>>>>>>>>>
>>>>>>>>>> some ideas, whats going on here?
>>>>>>>>>
>>>>>>>>> Can you get a stack trace from the core?
>>>>>>>>
>>>>>>>> I can give you an strace, see below. Other stack tools are currently not
>>>>>>>> available, because I need to compile them first for my yocto
>>>>>>>> environment. If you need something special please tell me.
>>>>>>>>
>>>>>>>>> This is Apache 2.4.x?
>>>>>>>>
>>>>>>>> yes it is Apache 2.4.16
>>>>>>>>
>>>>>>>>> Is it failing on a request or on startup?
>>>>>>>>
>>>>>>>> its failing on every https request.
>>>>>>>
>>>>>>> strace in this case isn't particularly helpful as it doesn't show where
>>>>>>> it is crashing.
>>>>>>>
>>>>>>> Can I see your nss.conf?
>>>>>>>
>>>>>>> What version of NSS are you using?
>>>>>>
>>>>>> I'am using nss in version 3.19.2
>>>>>>
>>>>>> here my nss.conf
>>>>>>
>>>>>> #
>>>>>> # This is the Apache server configuration file providing SSL support using.
>>>>>> # the mod_nss plugin. It contains the configuration directives to instruct
>>>>>> # the server how to serve pages over an https connection.
>>>>>> #
>>>>>> # Do NOT simply read the instructions in here without understanding
>>>>>> # what they do. They're here only as hints or reminders. If you are unsure
>>>>>> # consult the online docs. You have been warned.
>>>>>> #
>>>>>>
>>>>>> #
>>>>>> # When we also provide SSL we have to listen to the
>>>>>> # standard HTTP port (see above) and to the HTTPS port
>>>>>> #
>>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
>>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443"
>>>>>> #
>>>>>> Listen 443
>>>>>>
>>>>>> ##
>>>>>> ## SSL Global Context
>>>>>> ##
>>>>>> ## All SSL configuration in this context applies both to
>>>>>> ## the main server and all SSL-enabled virtual hosts.
>>>>>> ##
>>>>>>
>>>>>> #
>>>>>> # Some MIME-types for downloading Certificates and CRLs
>>>>>> #
>>>>>> AddType application/x-x509-ca-cert .crt
>>>>>> AddType application/x-pkcs7-crl .crl
>>>>>>
>>>>>>
>>>>>> # Pass Phrase Helper:
>>>>>> # This helper program stores the token password pins between
>>>>>> # restarts of Apache.
>>>>>> # Unfortunately the directive is required even if we use no password
>>>>>> # (though in such case the program should never be used)
>>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache
>>>>>>
>>>>>> # Pass Phrase Dialog:
>>>>>> # Configure the pass phrase gathering process.
>>>>>> # The filtering dialog program (`builtin' is a internal
>>>>>> # terminal dialog) has to provide the pass phrase on stdout.
>>>>>> #NSSPassPhraseDialog builtin
>>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf
>>>>>>
>>>>>> # Configure the SSL Session Cache.
>>>>>> # NSSSessionCacheSize is the number of entries in the cache.
>>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds).
>>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds).
>>>>>> NSSSessionCacheSize 10000
>>>>>> NSSSessionCacheTimeout 100
>>>>>> NSSSession3CacheTimeout 86400
>>>>>>
>>>>>> #RFC 5077
>>>>>> NSSSessionTickets off
>>>>>
>>>>> SessionTickets are on not off
>>>>>
>>>>> NSSSessionTickets on
>>>>>
>>>>
>>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss
>>>> parameters for every virtual host. It seems that the problem is here. Because
>>>> if I remove the include of httpd-vhosts.conf no segfaults occur.
>>>>
>>>> here my httpd-vhosts.conf with nss related parameters. Are these
>>>> parameters capable for a virtual host configuration?
>>>
>>> You've included some directives that are expected to be global:
>>>
>>> NSSPassPhraseHelper
>>> NSSPassPhraseDialog
>>> NSSSession*Cache*
>>> NSSCertificateDatabase
>>>
>>> I doubt they hurt anything.
>>
>> ok thx, I removed these directives from the httpd-vhosts.conf
>>
>>>
>>> I still can't duplicate a crash. I ran it through valgrind to see if
>>> there was a memory issue and came up with nothing too.
>>>
>>> This could be an architecture problem that I just can't see on x86
>>> hardware I suppose.
>>
>> I'am using a arm hardware (imx6)
>>
>>> This particular crash seems rather strange too. Within mod_nss enabling
>>> this just calls:
>>>
>>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS,
>>> mctx->sc->session_tickets);
>>>
>>> Basically enabling it in the model server socket. I would imagine the
>>> crash is probably deeper in NSS.
>>
>>
>>> To narrow things down you might try the NSS tool selfserv. It has an
>>> option, -u, to enable session tickets. It might eliminate mod_nss as the
>>> crash source (or it might implicate it too).
>>
>>
>> thx for the hint to the NSS selfserv tool.
>>
>> I'am using it with DSA Keys instead of Eliptic Curve Keys because
>> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers)
>>
>> First I generate a new nss db.
>>
>> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
>> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
>> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
>> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf
>> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf
>>
>> Then starting selfserv with params:
>>
>> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u
>>
>> selfserv: About to call accept.
>>
>> Now I perform a https request with the Chrome Browser
>>
>> GET /test.php HTTP/1.1
>> Host: 192.168.1.229
>> Connection: keep-alive
>> Cache-Control: max-age=0
>> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
>> Upgrade-Insecure-Requests: 1
>> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
>> DNT: 1
>> Accept-Encoding: gzip, deflate, sdch
>> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
>>
>> EOF
>>
>> it looks for me that it works with selfserv. But I'm not sure if the Session
>> Ticket works and if it makes a difference not using ECC Keys.
>
> I checked with wireshark, and I saw the TLS Session Ticket Extension
> in the hello packet there.
>
> I'also switched back to my old ciphers in my apache setup nss.conf
>
> NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha
>
> and this is working together with NSSSessionTickets on.
>
> So it breaks if i'am using ECC Ciphersuite together with Session Tickets!
>
> NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256
> NSSSessionTickets on
>
> is this something that can be explained?
Hmm, interesting. It gives me something to look at. I was in the middle
of responding to your previous message when this one came in.
I had been testing with an RSA key. Let me try an ECC key to see if I
can reproduce the crash (no success yet).
Here is the previous bit I was going to send:
This actually generated an RSA key for the cert you used for selfserv.
certutil will generate a certificate if the -k <id> option isn't provided.
You can do the same with ECC by adding to the last command -k ec -q <curve>
rob
More information about the Mod_nss-list
mailing list