From oliver.graute at gmail.com Mon Feb 22 12:54:34 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Mon, 22 Feb 2016 13:54:34 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log Message-ID: <20160222125434.GA24698@graute-opti> Hello, I installed the mod_nss plugin in version 1.0.12 on my apache webserver, TLS on Port 443 is working fine until I enable the new NSSSession ticket feature in my nss.conf with: #RFC 5077 NSSSessionTickets on then something is broken, I see segfaults in my apache error log: [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) and in Chrome Browser I got: ERR_SSL_VERSION_OR_CIPHER_MISMATCH I tested also a basic ssl client connection with openssl: openssl s_client -connect 192.168.1.229:443 -state -debug SSL_connect:SSLv3 read server certificate A SSL_connect:SSLv3 read server key exchange A SSL_connect:SSLv3 read server done A write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... SSL_connect:SSLv3 write client key exchange A write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) 0000 - 14 03 03 00 01 01 ...... SSL_connect:SSLv3 write change cipher spec A write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' SSL_connect:SSLv3 write finished A SSL_connect:SSLv3 flush data read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) SSL_connect:failed in SSLv3 read server session ticket A 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: apache and mod_nss are build from the sources for an embedded yocto environment. some ideas, whats going on here? Best regards, Oliver From rcritten at redhat.com Mon Feb 22 14:23:14 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 22 Feb 2016 09:23:14 -0500 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160222125434.GA24698@graute-opti> References: <20160222125434.GA24698@graute-opti> Message-ID: <56CB19D2.5010307@redhat.com> Oliver Graute wrote: > Hello, > > I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > TLS on Port 443 is working fine until I enable the new NSSSession ticket > feature in my nss.conf with: > > #RFC 5077 > NSSSessionTickets on > > then something is broken, I see segfaults in my apache error log: > > [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > > > and in Chrome Browser I got: > > ERR_SSL_VERSION_OR_CIPHER_MISMATCH > > I tested also a basic ssl client connection with openssl: > > openssl s_client -connect 192.168.1.229:443 -state -debug > > SSL_connect:SSLv3 read server certificate A > SSL_connect:SSLv3 read server key exchange A > SSL_connect:SSLv3 read server done A > write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > SSL_connect:SSLv3 write client key exchange A > write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > 0000 - 14 03 03 00 01 01 ...... > SSL_connect:SSLv3 write change cipher spec A > write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > SSL_connect:SSLv3 write finished A > SSL_connect:SSLv3 flush data > read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > SSL_connect:failed in SSLv3 read server session ticket A > 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > > apache and mod_nss are build from the sources for an embedded yocto environment. > > some ideas, whats going on here? Can you get a stack trace from the core? This is Apache 2.4.x? Is it failing on a request or on startup? rob From oliver.graute at gmail.com Tue Feb 23 08:58:26 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Tue, 23 Feb 2016 09:58:26 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56CB19D2.5010307@redhat.com> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> Message-ID: <20160223085826.GA11923@graute-opti> On 22/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > Hello, > > > > I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > > TLS on Port 443 is working fine until I enable the new NSSSession ticket > > feature in my nss.conf with: > > > > #RFC 5077 > > NSSSessionTickets on > > > > then something is broken, I see segfaults in my apache error log: > > > > [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > > [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > > [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > > [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > > [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > > [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > > [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > > [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > > [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > > [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > > [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > > [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > > [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > > > > > > and in Chrome Browser I got: > > > > ERR_SSL_VERSION_OR_CIPHER_MISMATCH > > > > I tested also a basic ssl client connection with openssl: > > > > openssl s_client -connect 192.168.1.229:443 -state -debug > > > > SSL_connect:SSLv3 read server certificate A > > SSL_connect:SSLv3 read server key exchange A > > SSL_connect:SSLv3 read server done A > > write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > > 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > > 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > > 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > > 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > > 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > > SSL_connect:SSLv3 write client key exchange A > > write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > > 0000 - 14 03 03 00 01 01 ...... > > SSL_connect:SSLv3 write change cipher spec A > > write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > > 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > > 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > > 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > > SSL_connect:SSLv3 write finished A > > SSL_connect:SSLv3 flush data > > read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > > SSL_connect:failed in SSLv3 read server session ticket A > > 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > > > > apache and mod_nss are build from the sources for an embedded yocto environment. > > > > some ideas, whats going on here? > > Can you get a stack trace from the core? I can give you an strace, see below. Other stack tools are currently not available, because I need to compile them first for my yocto environment. If you need something special please tell me. > This is Apache 2.4.x? yes it is Apache 2.4.16 > Is it failing on a request or on startup? its failing on every https request. [Fri Feb 19 11:31:13.701795 2016] [:info] [pid 507] Using nickname localhost - xxxxxxxxx root at xxx-box:/var/apache2/logs# strace -p 507 Process 507 attached poll([{fd=26, events=POLLIN}], 1, 20000) = 1 ([{fd=26, revents=POLLIN}]) read(26, "", 8000) = 0 gettimeofday({1455877950, 980711}, NULL) = 0 gettimeofday({1455877950, 982582}, NULL) = 0 gettimeofday({1455877950, 984651}, NULL) = 0 write(16, "[Fri Feb 19 11:32:30.984651 2016"..., 131) = 131 shutdown(26, SHUT_WR) = 0 poll([{fd=26, events=POLLIN}], 1, 2000) = 1 ([{fd=26, revents=POLLIN|POLLHUP}]) read(26, "", 512) = 0 close(26) = 0 read(11, 0x7eb28a93, 1) = -1 EAGAIN (Resource temporarily unavailable) gettimeofday({1455877950, 997674}, NULL) = 0 semop(294918, {{0, -1, SEM_UNDO}}, 1) = 0 poll([{fd=10, events=POLLIN}, {fd=8, events=POLLIN}, {fd=6, events=POLLIN}, {fd=4, events=POLLIN}], 4, 10000) = 1 ([{fd=10, revents=POLLIN}]) accept(10, {sa_family=AF_INET6, sin6_port=htons(35559), inet_pton(AF_INET6, "::ffff:192.168.1.98", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 26 fcntl64(26, F_GETFD) = 0 fcntl64(26, F_SETFD, FD_CLOEXEC) = 0 semop(294918, {{0, 1, SEM_UNDO}}, 1) = 0 gettimeofday({1455877951, 10882}, NULL) = 0 getsockname(26, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "::ffff:192.168.1.229", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 gettimeofday({1455877951, 14250}, NULL) = 0 write(16, "[Fri Feb 19 11:32:31.014250 2016"..., 136) = 136 fcntl64(26, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(26, F_SETFL, O_RDWR|O_NONBLOCK) = 0 gettimeofday({1455877951, 21367}, NULL) = 0 gettimeofday({1455877951, 23415}, NULL) = 0 read(26, "\26\3\1\0\332\1\0\0\326\3\3\271\37N\340\342fJ+f\366G\370\2118\200\177qtD\344\274"..., 8000) = 223 gettimeofday({1455877951, 26790}, NULL) = 0 gettimeofday({1455877951, 28739}, NULL) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 543716789}) = 0 gettimeofday({1455877951, 33213}, NULL) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 548681372}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 550682330}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 552976330}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 554695539}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 557096955}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 558818830}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 588420997}) = 0 writev(26, [{"\26\3\3\6\221\2\0\0W\3\3\236^\366\311\224IpJ\177\252.7\242\10\216\273\35\17\236@\256g\4\334\356\2"..., 8000) = 126 gettimeofday({1455877951, 83933}, NULL) = 0 gettimeofday({1455877951, 103060}, NULL) = 0 gettimeofday({1455877951, 103782}, NULL) = 0 gettimeofday({1455877951, 104490}, NULL) = 0 gettimeofday({1455877951, 105171}, NULL) = 0 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} --- chdir("/usr") = 0 rt_sigaction(SIGSEGV, {SIG_DFL, [], SA_RESTORER|SA_INTERRUPT, 0x76dfba90}, {SIG_DFL, [], SA_RESTORER|SA_RESETHAND, 0x76dfba90}, 8) = 0 kill(507, SIGSEGV) = 0 sigreturn({mask=[]}) = 11906264 --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_USER, si_pid=507, si_uid=1} --- +++ killed by SIGSEGV +++ Best regards, Oliver From oliver.graute at gmail.com Mon Feb 22 14:57:45 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Mon, 22 Feb 2016 15:57:45 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56CB19D2.5010307@redhat.com> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> Message-ID: <20160222145745.GA10896@graute-opti> On 22/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > Hello, > > > > I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > > TLS on Port 443 is working fine until I enable the new NSSSession ticket > > feature in my nss.conf with: > > > > #RFC 5077 > > NSSSessionTickets on > > > > then something is broken, I see segfaults in my apache error log: > > > > [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > > [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > > [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > > [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > > [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > > [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > > [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > > [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > > [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > > [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > > [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > > [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > > [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > > > > > > and in Chrome Browser I got: > > > > ERR_SSL_VERSION_OR_CIPHER_MISMATCH > > > > I tested also a basic ssl client connection with openssl: > > > > openssl s_client -connect 192.168.1.229:443 -state -debug > > > > SSL_connect:SSLv3 read server certificate A > > SSL_connect:SSLv3 read server key exchange A > > SSL_connect:SSLv3 read server done A > > write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > > 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > > 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > > 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > > 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > > 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > > SSL_connect:SSLv3 write client key exchange A > > write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > > 0000 - 14 03 03 00 01 01 ...... > > SSL_connect:SSLv3 write change cipher spec A > > write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > > 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > > 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > > 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > > SSL_connect:SSLv3 write finished A > > SSL_connect:SSLv3 flush data > > read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > > SSL_connect:failed in SSLv3 read server session ticket A > > 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > > > > apache and mod_nss are build from the sources for an embedded yocto environment. > > > > some ideas, whats going on here? > > Can you get a stack trace from the core? I can give you an strace, see below. Other stack tools are currently not available, because I need to compile them first for my yocto environment. If you need something special please tell me. > This is Apache 2.4.x? yes it is Apache 2.4.16 > Is it failing on a request or on startup? its failing on every https request. [Fri Feb 19 11:31:13.701795 2016] [:info] [pid 507] Using nickname localhost - xxxxxxxxx root at xxx-box:/var/apache2/logs# strace -p 507 Process 507 attached poll([{fd=26, events=POLLIN}], 1, 20000) = 1 ([{fd=26, revents=POLLIN}]) read(26, "", 8000) = 0 gettimeofday({1455877950, 980711}, NULL) = 0 gettimeofday({1455877950, 982582}, NULL) = 0 gettimeofday({1455877950, 984651}, NULL) = 0 write(16, "[Fri Feb 19 11:32:30.984651 2016"..., 131) = 131 shutdown(26, SHUT_WR) = 0 poll([{fd=26, events=POLLIN}], 1, 2000) = 1 ([{fd=26, revents=POLLIN|POLLHUP}]) read(26, "", 512) = 0 close(26) = 0 read(11, 0x7eb28a93, 1) = -1 EAGAIN (Resource temporarily unavailable) gettimeofday({1455877950, 997674}, NULL) = 0 semop(294918, {{0, -1, SEM_UNDO}}, 1) = 0 poll([{fd=10, events=POLLIN}, {fd=8, events=POLLIN}, {fd=6, events=POLLIN}, {fd=4, events=POLLIN}], 4, 10000) = 1 ([{fd=10, revents=POLLIN}]) accept(10, {sa_family=AF_INET6, sin6_port=htons(35559), inet_pton(AF_INET6, "::ffff:192.168.1.98", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 26 fcntl64(26, F_GETFD) = 0 fcntl64(26, F_SETFD, FD_CLOEXEC) = 0 semop(294918, {{0, 1, SEM_UNDO}}, 1) = 0 gettimeofday({1455877951, 10882}, NULL) = 0 getsockname(26, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "::ffff:192.168.1.229", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 gettimeofday({1455877951, 14250}, NULL) = 0 write(16, "[Fri Feb 19 11:32:31.014250 2016"..., 136) = 136 fcntl64(26, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(26, F_SETFL, O_RDWR|O_NONBLOCK) = 0 gettimeofday({1455877951, 21367}, NULL) = 0 gettimeofday({1455877951, 23415}, NULL) = 0 read(26, "\26\3\1\0\332\1\0\0\326\3\3\271\37N\340\342fJ+f\366G\370\2118\200\177qtD\344\274"..., 8000) = 223 gettimeofday({1455877951, 26790}, NULL) = 0 gettimeofday({1455877951, 28739}, NULL) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 543716789}) = 0 gettimeofday({1455877951, 33213}, NULL) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 548681372}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 550682330}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 552976330}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 554695539}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 557096955}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 558818830}) = 0 clock_gettime(CLOCK_MONOTONIC, {2079, 588420997}) = 0 writev(26, [{"\26\3\3\6\221\2\0\0W\3\3\236^\366\311\224IpJ\177\252.7\242\10\216\273\35\17\236@\256g\4\334\356\2"..., 8000) = 126 gettimeofday({1455877951, 83933}, NULL) = 0 gettimeofday({1455877951, 103060}, NULL) = 0 gettimeofday({1455877951, 103782}, NULL) = 0 gettimeofday({1455877951, 104490}, NULL) = 0 gettimeofday({1455877951, 105171}, NULL) = 0 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} --- chdir("/usr") = 0 rt_sigaction(SIGSEGV, {SIG_DFL, [], SA_RESTORER|SA_INTERRUPT, 0x76dfba90}, {SIG_DFL, [], SA_RESTORER|SA_RESETHAND, 0x76dfba90}, 8) = 0 kill(507, SIGSEGV) = 0 sigreturn({mask=[]}) = 11906264 --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_USER, si_pid=507, si_uid=1} --- +++ killed by SIGSEGV +++ Best regards, Oliver From rcritten at redhat.com Tue Feb 23 13:43:30 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 23 Feb 2016 08:43:30 -0500 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160223085826.GA11923@graute-opti> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> Message-ID: <56CC6202.9030700@redhat.com> Oliver Graute wrote: > On 22/02/16, Rob Crittenden wrote: >> Oliver Graute wrote: >>> Hello, >>> >>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, >>> TLS on Port 443 is working fine until I enable the new NSSSession ticket >>> feature in my nss.conf with: >>> >>> #RFC 5077 >>> NSSSessionTickets on >>> >>> then something is broken, I see segfaults in my apache error log: >>> >>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations >>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 >>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' >>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) >>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled >>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy >>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) >>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled >>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy >>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) >>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled >>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy >>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) >>> >>> >>> and in Chrome Browser I got: >>> >>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH >>> >>> I tested also a basic ssl client connection with openssl: >>> >>> openssl s_client -connect 192.168.1.229:443 -state -debug >>> >>> SSL_connect:SSLv3 read server certificate A >>> SSL_connect:SSLv3 read server key exchange A >>> SSL_connect:SSLv3 read server done A >>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) >>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE >>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... >>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... >>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... >>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... >>> SSL_connect:SSLv3 write client key exchange A >>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) >>> 0000 - 14 03 03 00 01 01 ...... >>> SSL_connect:SSLv3 write change cipher spec A >>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) >>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. >>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ >>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' >>> SSL_connect:SSLv3 write finished A >>> SSL_connect:SSLv3 flush data >>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) >>> SSL_connect:failed in SSLv3 read server session ticket A >>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: >>> >>> apache and mod_nss are build from the sources for an embedded yocto environment. >>> >>> some ideas, whats going on here? >> >> Can you get a stack trace from the core? > > I can give you an strace, see below. Other stack tools are currently not > available, because I need to compile them first for my yocto > environment. If you need something special please tell me. > >> This is Apache 2.4.x? > > yes it is Apache 2.4.16 > >> Is it failing on a request or on startup? > > its failing on every https request. strace in this case isn't particularly helpful as it doesn't show where it is crashing. Can I see your nss.conf? What version of NSS are you using? rob From oliver.graute at gmail.com Tue Feb 23 16:10:09 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Tue, 23 Feb 2016 17:10:09 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56CC6202.9030700@redhat.com> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> Message-ID: <20160223161009.GA8558@graute-opti> On 23/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > On 22/02/16, Rob Crittenden wrote: > >> Oliver Graute wrote: > >>> Hello, > >>> > >>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > >>> TLS on Port 443 is working fine until I enable the new NSSSession ticket > >>> feature in my nss.conf with: > >>> > >>> #RFC 5077 > >>> NSSSessionTickets on > >>> > >>> then something is broken, I see segfaults in my apache error log: > >>> > >>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > >>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > >>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > >>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > >>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > >>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > >>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > >>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > >>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > >>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > >>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > >>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > >>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > >>> > >>> > >>> and in Chrome Browser I got: > >>> > >>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH > >>> > >>> I tested also a basic ssl client connection with openssl: > >>> > >>> openssl s_client -connect 192.168.1.229:443 -state -debug > >>> > >>> SSL_connect:SSLv3 read server certificate A > >>> SSL_connect:SSLv3 read server key exchange A > >>> SSL_connect:SSLv3 read server done A > >>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > >>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > >>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > >>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > >>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > >>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > >>> SSL_connect:SSLv3 write client key exchange A > >>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > >>> 0000 - 14 03 03 00 01 01 ...... > >>> SSL_connect:SSLv3 write change cipher spec A > >>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > >>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > >>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > >>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > >>> SSL_connect:SSLv3 write finished A > >>> SSL_connect:SSLv3 flush data > >>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > >>> SSL_connect:failed in SSLv3 read server session ticket A > >>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > >>> > >>> apache and mod_nss are build from the sources for an embedded yocto environment. > >>> > >>> some ideas, whats going on here? > >> > >> Can you get a stack trace from the core? > > > > I can give you an strace, see below. Other stack tools are currently not > > available, because I need to compile them first for my yocto > > environment. If you need something special please tell me. > > > >> This is Apache 2.4.x? > > > > yes it is Apache 2.4.16 > > > >> Is it failing on a request or on startup? > > > > its failing on every https request. > > strace in this case isn't particularly helpful as it doesn't show where > it is crashing. > > Can I see your nss.conf? > > What version of NSS are you using? I'am using nss in version 3.19.2 here my nss.conf # # This is the Apache server configuration file providing SSL support using. # the mod_nss plugin. It contains the configuration directives to instruct # the server how to serve pages over an https connection. # # Do NOT simply read the instructions in here without understanding # what they do. They're here only as hints or reminders. If you are unsure # consult the online docs. You have been warned. # # # When we also provide SSL we have to listen to the # standard HTTP port (see above) and to the HTTPS port # # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" # Listen 443 ## ## SSL Global Context ## ## All SSL configuration in this context applies both to ## the main server and all SSL-enabled virtual hosts. ## # # Some MIME-types for downloading Certificates and CRLs # AddType application/x-x509-ca-cert .crt AddType application/x-pkcs7-crl .crl # Pass Phrase Helper: # This helper program stores the token password pins between # restarts of Apache. # Unfortunately the directive is required even if we use no password # (though in such case the program should never be used) NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache # Pass Phrase Dialog: # Configure the pass phrase gathering process. # The filtering dialog program (`builtin' is a internal # terminal dialog) has to provide the pass phrase on stdout. #NSSPassPhraseDialog builtin NSSPassPhraseDialog file:/etc/apache2/password.conf # Configure the SSL Session Cache. # NSSSessionCacheSize is the number of entries in the cache. # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). NSSSessionCacheSize 10000 NSSSessionCacheTimeout 100 NSSSession3CacheTimeout 86400 #RFC 5077 NSSSessionTickets off # # Pseudo Random Number Generator (PRNG): # Configure one or more sources to seed the PRNG of the SSL library. # The seed data should be of good random quality. # WARNING! On some platforms /dev/random blocks if not enough entropy # is available. Those platforms usually also provide a non-blocking # device, /dev/urandom, which may be used instead. # # This does not support seeding the RNG with each connection. NSSRandomSeed startup builtin #NSSRandomSeed startup file:/dev/random 512 #NSSRandomSeed startup file:/dev/urandom 512 # # TLS Negotiation configuration under RFC 5746 # # Only renegotiate if the peer's hello bears the TLS renegotiation_info # extension. Default off. NSSRenegotiation off # Peer must send Signaling Cipher Suite Value (SCSV) or # Renegotiation Info (RI) extension in ALL handshakes. Default: off NSSRequireSafeNegotiation off ## ## SSL Virtual Host Context ## ServerAdmin webmaster at localhost DocumentRoot "/var/www" Options FollowSymLinks Options Indexes FollowSymLinks MultiViews AllowOverride None Require all granted # Default syslog facility is local7. ErrorLog "/var/apache2/logs/nss_log" LogLevel debug # SSL Engine Switch: # Enable/Disable SSL for this virtual host. NSSEngine on # Enable or disables Server Name Identification (SN) extension check for SSL. NSSSNI off # SSL Cipher Suite: # List the ciphers that the client is permitted to negotiate. # See the mod_nss documentation for a complete list. NSSProtocol TLSv1.2 #NSSProtocol TLSv1.0,TLSv1.1,TLSv1.2 NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha_256,+ecdhe_ecdsa_aes_128_sha_256 # SSL Certificate Nickname: # The nickname of the RSA server certificate you are going to use. NSSNickname "localhost - xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # SSL Certificate Nickname: # The nickname of the ECC server certificate you are going to use, if you # have an ECC-enabled version of NSS and mod_nss #NSSECCNickname "Safir:ClefPrivK" NSSEnforceValidCerts off # Server Certificate Database: # The NSS security database directory that holds the certificates and # keys. The database consists of 3 files: cert9.db, key4.db and pkcs11.txt # Provide the directory where these files exist. NSSCertificateDatabase /etc/apache2/nss-conf # Client Authentication (Type): # Client certificate verification type. Types are none, optional and # require. #NSSVerifyClient require # SSL Engine Options: # Set various options for the SSL engine. # o FakeBasicAuth: # Translate the client X.509 into a Basic Authorisation. This means that # the standard Auth/DBMAuth methods can be used for access control. The # user name is the `one line' version of the client's X.509 certificate. # Note that no password is obtained from the user. Every entry in the user # file needs this password: `xxj31ZMTZzkVA'. # o ExportCertData: # This exports two additional environment variables: SSL_CLIENT_CERT and # SSL_SERVER_CERT. These contain the PEM-encoded certificates of the # server (always existing) and the client (only existing when client # authentication is used). This can be used to import the certificates # into CGI scripts. # o StdEnvVars: # This exports the standard SSL/TLS related `SSL_*' environment variables. # Per default this exportation is switched off for performance reasons, # because the extraction step is an expensive operation and is usually # useless for serving static content. So one usually enables the # exportation for CGI and SSI requests only. # o StrictRequire: # This denies access when "NSSRequireSSL" or "NSSRequire" applied even # under a "Satisfy any" situation, i.e. when it applies access is denied # and no other module can change it. # o OptRenegotiate: # This enables optimized SSL connection renegotiation handling when SSL # directives are used in per-directory context. NSSOptions +FakeBasicAuth +ExportCertData +CompatEnvVars +StrictRequire NSSOptions +StdEnvVars NSSOptions +StdEnvVars Best regards, Oliver From oliver.graute at gmail.com Tue Feb 23 19:05:37 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Tue, 23 Feb 2016 20:05:37 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160223161009.GA8558@graute-opti> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> Message-ID: On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: > On 23/02/16, Rob Crittenden wrote: >> Oliver Graute wrote: >> > On 22/02/16, Rob Crittenden wrote: >> >> Oliver Graute wrote: >> >>> Hello, >> >>> >> >>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, >> >>> TLS on Port 443 is working fine until I enable the new NSSSession ticket >> >>> feature in my nss.conf with: >> >>> >> >>> #RFC 5077 >> >>> NSSSessionTickets on >> >>> >> >>> then something is broken, I see segfaults in my apache error log: >> >>> >> >>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations >> >>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 >> >>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' >> >>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) >> >>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled >> >>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy >> >>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) >> >>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled >> >>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy >> >>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) >> >>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled >> >>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy >> >>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) >> >>> >> >>> >> >>> and in Chrome Browser I got: >> >>> >> >>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH >> >>> >> >>> I tested also a basic ssl client connection with openssl: >> >>> >> >>> openssl s_client -connect 192.168.1.229:443 -state -debug >> >>> >> >>> SSL_connect:SSLv3 read server certificate A >> >>> SSL_connect:SSLv3 read server key exchange A >> >>> SSL_connect:SSLv3 read server done A >> >>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) >> >>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE >> >>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... >> >>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... >> >>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... >> >>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... >> >>> SSL_connect:SSLv3 write client key exchange A >> >>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) >> >>> 0000 - 14 03 03 00 01 01 ...... >> >>> SSL_connect:SSLv3 write change cipher spec A >> >>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) >> >>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. >> >>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ >> >>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' >> >>> SSL_connect:SSLv3 write finished A >> >>> SSL_connect:SSLv3 flush data >> >>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) >> >>> SSL_connect:failed in SSLv3 read server session ticket A >> >>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: >> >>> >> >>> apache and mod_nss are build from the sources for an embedded yocto environment. >> >>> >> >>> some ideas, whats going on here? >> >> >> >> Can you get a stack trace from the core? >> > >> > I can give you an strace, see below. Other stack tools are currently not >> > available, because I need to compile them first for my yocto >> > environment. If you need something special please tell me. >> > >> >> This is Apache 2.4.x? >> > >> > yes it is Apache 2.4.16 >> > >> >> Is it failing on a request or on startup? >> > >> > its failing on every https request. >> >> strace in this case isn't particularly helpful as it doesn't show where >> it is crashing. >> >> Can I see your nss.conf? >> >> What version of NSS are you using? > > I'am using nss in version 3.19.2 > > here my nss.conf > > # > # This is the Apache server configuration file providing SSL support using. > # the mod_nss plugin. It contains the configuration directives to instruct > # the server how to serve pages over an https connection. > # > # Do NOT simply read the instructions in here without understanding > # what they do. They're here only as hints or reminders. If you are unsure > # consult the online docs. You have been warned. > # > > # > # When we also provide SSL we have to listen to the > # standard HTTP port (see above) and to the HTTPS port > # > # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two > # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" > # > Listen 443 > > ## > ## SSL Global Context > ## > ## All SSL configuration in this context applies both to > ## the main server and all SSL-enabled virtual hosts. > ## > > # > # Some MIME-types for downloading Certificates and CRLs > # > AddType application/x-x509-ca-cert .crt > AddType application/x-pkcs7-crl .crl > > > # Pass Phrase Helper: > # This helper program stores the token password pins between > # restarts of Apache. > # Unfortunately the directive is required even if we use no password > # (though in such case the program should never be used) > NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache > > # Pass Phrase Dialog: > # Configure the pass phrase gathering process. > # The filtering dialog program (`builtin' is a internal > # terminal dialog) has to provide the pass phrase on stdout. > #NSSPassPhraseDialog builtin > NSSPassPhraseDialog file:/etc/apache2/password.conf > > # Configure the SSL Session Cache. > # NSSSessionCacheSize is the number of entries in the cache. > # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). > # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). > NSSSessionCacheSize 10000 > NSSSessionCacheTimeout 100 > NSSSession3CacheTimeout 86400 > > #RFC 5077 > NSSSessionTickets off SessionTickets are on not off NSSSessionTickets on Best regards, Oliver From oliver.graute at gmail.com Wed Feb 24 10:02:16 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Wed, 24 Feb 2016 11:02:16 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> Message-ID: <20160224100216.GA29619@graute-opti> On 23/02/16, Oliver Graute wrote: > On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: > > On 23/02/16, Rob Crittenden wrote: > >> Oliver Graute wrote: > >> > On 22/02/16, Rob Crittenden wrote: > >> >> Oliver Graute wrote: > >> >>> Hello, > >> >>> > >> >>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > >> >>> TLS on Port 443 is working fine until I enable the new NSSSession ticket > >> >>> feature in my nss.conf with: > >> >>> > >> >>> #RFC 5077 > >> >>> NSSSessionTickets on > >> >>> > >> >>> then something is broken, I see segfaults in my apache error log: > >> >>> > >> >>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > >> >>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > >> >>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > >> >>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > >> >>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > >> >>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > >> >>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > >> >>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > >> >>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > >> >>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > >> >>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > >> >>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > >> >>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > >> >>> > >> >>> > >> >>> and in Chrome Browser I got: > >> >>> > >> >>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH > >> >>> > >> >>> I tested also a basic ssl client connection with openssl: > >> >>> > >> >>> openssl s_client -connect 192.168.1.229:443 -state -debug > >> >>> > >> >>> SSL_connect:SSLv3 read server certificate A > >> >>> SSL_connect:SSLv3 read server key exchange A > >> >>> SSL_connect:SSLv3 read server done A > >> >>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > >> >>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > >> >>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > >> >>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > >> >>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > >> >>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > >> >>> SSL_connect:SSLv3 write client key exchange A > >> >>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > >> >>> 0000 - 14 03 03 00 01 01 ...... > >> >>> SSL_connect:SSLv3 write change cipher spec A > >> >>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > >> >>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > >> >>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > >> >>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > >> >>> SSL_connect:SSLv3 write finished A > >> >>> SSL_connect:SSLv3 flush data > >> >>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > >> >>> SSL_connect:failed in SSLv3 read server session ticket A > >> >>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > >> >>> > >> >>> apache and mod_nss are build from the sources for an embedded yocto environment. > >> >>> > >> >>> some ideas, whats going on here? > >> >> > >> >> Can you get a stack trace from the core? > >> > > >> > I can give you an strace, see below. Other stack tools are currently not > >> > available, because I need to compile them first for my yocto > >> > environment. If you need something special please tell me. > >> > > >> >> This is Apache 2.4.x? > >> > > >> > yes it is Apache 2.4.16 > >> > > >> >> Is it failing on a request or on startup? > >> > > >> > its failing on every https request. > >> > >> strace in this case isn't particularly helpful as it doesn't show where > >> it is crashing. > >> > >> Can I see your nss.conf? > >> > >> What version of NSS are you using? > > > > I'am using nss in version 3.19.2 > > > > here my nss.conf > > > > # > > # This is the Apache server configuration file providing SSL support using. > > # the mod_nss plugin. It contains the configuration directives to instruct > > # the server how to serve pages over an https connection. > > # > > # Do NOT simply read the instructions in here without understanding > > # what they do. They're here only as hints or reminders. If you are unsure > > # consult the online docs. You have been warned. > > # > > > > # > > # When we also provide SSL we have to listen to the > > # standard HTTP port (see above) and to the HTTPS port > > # > > # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two > > # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" > > # > > Listen 443 > > > > ## > > ## SSL Global Context > > ## > > ## All SSL configuration in this context applies both to > > ## the main server and all SSL-enabled virtual hosts. > > ## > > > > # > > # Some MIME-types for downloading Certificates and CRLs > > # > > AddType application/x-x509-ca-cert .crt > > AddType application/x-pkcs7-crl .crl > > > > > > # Pass Phrase Helper: > > # This helper program stores the token password pins between > > # restarts of Apache. > > # Unfortunately the directive is required even if we use no password > > # (though in such case the program should never be used) > > NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache > > > > # Pass Phrase Dialog: > > # Configure the pass phrase gathering process. > > # The filtering dialog program (`builtin' is a internal > > # terminal dialog) has to provide the pass phrase on stdout. > > #NSSPassPhraseDialog builtin > > NSSPassPhraseDialog file:/etc/apache2/password.conf > > > > # Configure the SSL Session Cache. > > # NSSSessionCacheSize is the number of entries in the cache. > > # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). > > # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). > > NSSSessionCacheSize 10000 > > NSSSessionCacheTimeout 100 > > NSSSession3CacheTimeout 86400 > > > > #RFC 5077 > > NSSSessionTickets off > > SessionTickets are on not off > > NSSSessionTickets on > in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss parameters for every virtual host. It seems that the problem is here. Because if I remove the include of httpd-vhosts.conf no segfaults occur. here my httpd-vhosts.conf with nss related parameters. Are these parameters capable for a virtual host configuration? Best regards, Oliver # Virtual Hosts # # Required modules: mod_log_config # If you want to maintain multiple domains/hostnames on your # machine you can setup VirtualHost containers for them. Most configurations # use only name-based virtual hosts so the server doesn't need to worry about # IP addresses. This is indicated by the asterisks in the directives below. # # Please see the documentation at # # for further details before you try to setup virtual hosts. # # You may use the command line option '-S' to verify your virtual host # configuration. # # VirtualHost example: # Almost any Apache directive may go into a VirtualHost container. # The first VirtualHost section is used for all requests that do not # match a ServerName or ServerAlias in any block. # ServerAdmin webmaster at example.com DocumentRoot "/var/www" Options FollowSymLinks Options Indexes FollowSymLinks MultiViews AllowOverride None Require all granted ServerName www.example.com ServerAlias www.example.com ErrorLog "/var/apache2/logs/example.com-error_log" CustomLog "/var/apache2/logs/example.com-access_log" common LogLevel debug AddType application/x-x509-ca-cert .crt AddType application/x-pkcs7-crl .crl NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache NSSPassPhraseDialog file:/etc/apache2/password.conf NSSSessionCacheSize 10000 NSSSessionCacheTimeout 100 NSSSession3CacheTimeout 86400 NSSSessionTickets on NSSRenegotiation off NSSRequireSafeNegotiation off NSSEngine on NSSSNI off NSSProtocol TLSv1.2 NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha_256,+ecdhe_ecdsa_aes_128_sha_256 NSSNickname "localhost - XXXXXX" NSSEnforceValidCerts off NSSCertificateDatabase /etc/apache2/nss-conf NSSOptions +FakeBasicAuth +ExportCertData +CompatEnvVars +StrictRequire NSSOptions +StdEnvVars NSSOptions +StdEnvVars Listen 444 ServerAdmin webmaster at example2.com DocumentRoot "/var/www/example2.com" Options FollowSymLinks Options Indexes FollowSymLinks MultiViews AllowOverride None Require all granted ServerName www.example2.com ErrorLog "/var/apache2/logs/example2.com-error_log" CustomLog "/var/apache2/logs/example2.com-access_log" common LogLevel debug AddType application/x-x509-ca-cert .crt AddType application/x-pkcs7-crl .crl NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache NSSPassPhraseDialog file:/etc/apache2/password.conf NSSSessionCacheSize 10000 NSSSessionCacheTimeout 100 NSSSession3CacheTimeout 86400 NSSSessionTickets on NSSRenegotiation off NSSRequireSafeNegotiation off NSSEngine on NSSSNI off NSSProtocol TLSv1.2 NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha_256,+ecdhe_ecdsa_aes_128_sha_256 NSSNickname "localhost - XXXXXX" NSSEnforceValidCerts off NSSCertificateDatabase /etc/apache2/nss-conf NSSOptions +FakeBasicAuth +ExportCertData +CompatEnvVars +StrictRequire NSSOptions +StdEnvVars NSSOptions +StdEnvVars Listen 1226 ServerAdmin webmaster at example3.com DocumentRoot "/var/www/iMTR" Options FollowSymLinks Options Indexes FollowSymLinks MultiViews AllowOverride None Require all granted ServerName www.example3.com ErrorLog "/var/apache2/logs/iMTR-error_log" CustomLog "/var/apache2/logs/iMTR-access_log" common LogLevel debug AddType application/x-x509-ca-cert .crt AddType application/x-pkcs7-crl .crl NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache NSSPassPhraseDialog file:/etc/apache2/password.conf NSSSessionCacheSize 10000 NSSSessionCacheTimeout 100 NSSSession3CacheTimeout 86400 NSSSessionTickets on NSSRenegotiation off NSSRequireSafeNegotiation off NSSEngine on NSSSNI off NSSProtocol TLSv1.2 NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha_256,+ecdhe_ecdsa_aes_128_sha_256 NSSNickname "localhost - XXXXXX" NSSEnforceValidCerts off NSSCertificateDatabase /etc/apache2/nss-conf NSSOptions +FakeBasicAuth +ExportCertData +CompatEnvVars +StrictRequire NSSOptions +StdEnvVars NSSOptions +StdEnvVars From rcritten at redhat.com Wed Feb 24 16:21:27 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 24 Feb 2016 11:21:27 -0500 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160224100216.GA29619@graute-opti> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> Message-ID: <56CDD887.1000101@redhat.com> Oliver Graute wrote: > On 23/02/16, Oliver Graute wrote: >> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: >>> On 23/02/16, Rob Crittenden wrote: >>>> Oliver Graute wrote: >>>>> On 22/02/16, Rob Crittenden wrote: >>>>>> Oliver Graute wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, >>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket >>>>>>> feature in my nss.conf with: >>>>>>> >>>>>>> #RFC 5077 >>>>>>> NSSSessionTickets on >>>>>>> >>>>>>> then something is broken, I see segfaults in my apache error log: >>>>>>> >>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations >>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 >>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' >>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) >>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled >>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy >>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) >>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled >>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy >>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) >>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled >>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy >>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) >>>>>>> >>>>>>> >>>>>>> and in Chrome Browser I got: >>>>>>> >>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH >>>>>>> >>>>>>> I tested also a basic ssl client connection with openssl: >>>>>>> >>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug >>>>>>> >>>>>>> SSL_connect:SSLv3 read server certificate A >>>>>>> SSL_connect:SSLv3 read server key exchange A >>>>>>> SSL_connect:SSLv3 read server done A >>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) >>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE >>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... >>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... >>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... >>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... >>>>>>> SSL_connect:SSLv3 write client key exchange A >>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) >>>>>>> 0000 - 14 03 03 00 01 01 ...... >>>>>>> SSL_connect:SSLv3 write change cipher spec A >>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) >>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. >>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ >>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' >>>>>>> SSL_connect:SSLv3 write finished A >>>>>>> SSL_connect:SSLv3 flush data >>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) >>>>>>> SSL_connect:failed in SSLv3 read server session ticket A >>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: >>>>>>> >>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. >>>>>>> >>>>>>> some ideas, whats going on here? >>>>>> >>>>>> Can you get a stack trace from the core? >>>>> >>>>> I can give you an strace, see below. Other stack tools are currently not >>>>> available, because I need to compile them first for my yocto >>>>> environment. If you need something special please tell me. >>>>> >>>>>> This is Apache 2.4.x? >>>>> >>>>> yes it is Apache 2.4.16 >>>>> >>>>>> Is it failing on a request or on startup? >>>>> >>>>> its failing on every https request. >>>> >>>> strace in this case isn't particularly helpful as it doesn't show where >>>> it is crashing. >>>> >>>> Can I see your nss.conf? >>>> >>>> What version of NSS are you using? >>> >>> I'am using nss in version 3.19.2 >>> >>> here my nss.conf >>> >>> # >>> # This is the Apache server configuration file providing SSL support using. >>> # the mod_nss plugin. It contains the configuration directives to instruct >>> # the server how to serve pages over an https connection. >>> # >>> # Do NOT simply read the instructions in here without understanding >>> # what they do. They're here only as hints or reminders. If you are unsure >>> # consult the online docs. You have been warned. >>> # >>> >>> # >>> # When we also provide SSL we have to listen to the >>> # standard HTTP port (see above) and to the HTTPS port >>> # >>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two >>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" >>> # >>> Listen 443 >>> >>> ## >>> ## SSL Global Context >>> ## >>> ## All SSL configuration in this context applies both to >>> ## the main server and all SSL-enabled virtual hosts. >>> ## >>> >>> # >>> # Some MIME-types for downloading Certificates and CRLs >>> # >>> AddType application/x-x509-ca-cert .crt >>> AddType application/x-pkcs7-crl .crl >>> >>> >>> # Pass Phrase Helper: >>> # This helper program stores the token password pins between >>> # restarts of Apache. >>> # Unfortunately the directive is required even if we use no password >>> # (though in such case the program should never be used) >>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache >>> >>> # Pass Phrase Dialog: >>> # Configure the pass phrase gathering process. >>> # The filtering dialog program (`builtin' is a internal >>> # terminal dialog) has to provide the pass phrase on stdout. >>> #NSSPassPhraseDialog builtin >>> NSSPassPhraseDialog file:/etc/apache2/password.conf >>> >>> # Configure the SSL Session Cache. >>> # NSSSessionCacheSize is the number of entries in the cache. >>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). >>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). >>> NSSSessionCacheSize 10000 >>> NSSSessionCacheTimeout 100 >>> NSSSession3CacheTimeout 86400 >>> >>> #RFC 5077 >>> NSSSessionTickets off >> >> SessionTickets are on not off >> >> NSSSessionTickets on >> > > in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss > parameters for every virtual host. It seems that the problem is here. Because > if I remove the include of httpd-vhosts.conf no segfaults occur. > > here my httpd-vhosts.conf with nss related parameters. Are these > parameters capable for a virtual host configuration? You've included some directives that are expected to be global: NSSPassPhraseHelper NSSPassPhraseDialog NSSSession*Cache* NSSCertificateDatabase I doubt they hurt anything. I still can't duplicate a crash. I ran it through valgrind to see if there was a memory issue and came up with nothing too. This could be an architecture problem that I just can't see on x86 hardware I suppose. This particular crash seems rather strange too. Within mod_nss enabling this just calls: SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, mctx->sc->session_tickets); Basically enabling it in the model server socket. I would imagine the crash is probably deeper in NSS. To narrow things down you might try the NSS tool selfserv. It has an option, -u, to enable session tickets. It might eliminate mod_nss as the crash source (or it might implicate it too). rob From oliver.graute at gmail.com Thu Feb 25 14:10:39 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Thu, 25 Feb 2016 15:10:39 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56CDD887.1000101@redhat.com> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> Message-ID: <20160225141039.GB29619@graute-opti> On 24/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > On 23/02/16, Oliver Graute wrote: > >> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: > >>> On 23/02/16, Rob Crittenden wrote: > >>>> Oliver Graute wrote: > >>>>> On 22/02/16, Rob Crittenden wrote: > >>>>>> Oliver Graute wrote: > >>>>>>> Hello, > >>>>>>> > >>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > >>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket > >>>>>>> feature in my nss.conf with: > >>>>>>> > >>>>>>> #RFC 5077 > >>>>>>> NSSSessionTickets on > >>>>>>> > >>>>>>> then something is broken, I see segfaults in my apache error log: > >>>>>>> > >>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > >>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > >>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > >>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > >>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > >>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > >>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > >>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > >>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > >>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > >>>>>>> > >>>>>>> > >>>>>>> and in Chrome Browser I got: > >>>>>>> > >>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH > >>>>>>> > >>>>>>> I tested also a basic ssl client connection with openssl: > >>>>>>> > >>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug > >>>>>>> > >>>>>>> SSL_connect:SSLv3 read server certificate A > >>>>>>> SSL_connect:SSLv3 read server key exchange A > >>>>>>> SSL_connect:SSLv3 read server done A > >>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > >>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > >>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > >>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > >>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > >>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > >>>>>>> SSL_connect:SSLv3 write client key exchange A > >>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > >>>>>>> 0000 - 14 03 03 00 01 01 ...... > >>>>>>> SSL_connect:SSLv3 write change cipher spec A > >>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > >>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > >>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > >>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > >>>>>>> SSL_connect:SSLv3 write finished A > >>>>>>> SSL_connect:SSLv3 flush data > >>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > >>>>>>> SSL_connect:failed in SSLv3 read server session ticket A > >>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > >>>>>>> > >>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. > >>>>>>> > >>>>>>> some ideas, whats going on here? > >>>>>> > >>>>>> Can you get a stack trace from the core? > >>>>> > >>>>> I can give you an strace, see below. Other stack tools are currently not > >>>>> available, because I need to compile them first for my yocto > >>>>> environment. If you need something special please tell me. > >>>>> > >>>>>> This is Apache 2.4.x? > >>>>> > >>>>> yes it is Apache 2.4.16 > >>>>> > >>>>>> Is it failing on a request or on startup? > >>>>> > >>>>> its failing on every https request. > >>>> > >>>> strace in this case isn't particularly helpful as it doesn't show where > >>>> it is crashing. > >>>> > >>>> Can I see your nss.conf? > >>>> > >>>> What version of NSS are you using? > >>> > >>> I'am using nss in version 3.19.2 > >>> > >>> here my nss.conf > >>> > >>> # > >>> # This is the Apache server configuration file providing SSL support using. > >>> # the mod_nss plugin. It contains the configuration directives to instruct > >>> # the server how to serve pages over an https connection. > >>> # > >>> # Do NOT simply read the instructions in here without understanding > >>> # what they do. They're here only as hints or reminders. If you are unsure > >>> # consult the online docs. You have been warned. > >>> # > >>> > >>> # > >>> # When we also provide SSL we have to listen to the > >>> # standard HTTP port (see above) and to the HTTPS port > >>> # > >>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two > >>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" > >>> # > >>> Listen 443 > >>> > >>> ## > >>> ## SSL Global Context > >>> ## > >>> ## All SSL configuration in this context applies both to > >>> ## the main server and all SSL-enabled virtual hosts. > >>> ## > >>> > >>> # > >>> # Some MIME-types for downloading Certificates and CRLs > >>> # > >>> AddType application/x-x509-ca-cert .crt > >>> AddType application/x-pkcs7-crl .crl > >>> > >>> > >>> # Pass Phrase Helper: > >>> # This helper program stores the token password pins between > >>> # restarts of Apache. > >>> # Unfortunately the directive is required even if we use no password > >>> # (though in such case the program should never be used) > >>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache > >>> > >>> # Pass Phrase Dialog: > >>> # Configure the pass phrase gathering process. > >>> # The filtering dialog program (`builtin' is a internal > >>> # terminal dialog) has to provide the pass phrase on stdout. > >>> #NSSPassPhraseDialog builtin > >>> NSSPassPhraseDialog file:/etc/apache2/password.conf > >>> > >>> # Configure the SSL Session Cache. > >>> # NSSSessionCacheSize is the number of entries in the cache. > >>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). > >>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). > >>> NSSSessionCacheSize 10000 > >>> NSSSessionCacheTimeout 100 > >>> NSSSession3CacheTimeout 86400 > >>> > >>> #RFC 5077 > >>> NSSSessionTickets off > >> > >> SessionTickets are on not off > >> > >> NSSSessionTickets on > >> > > > > in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss > > parameters for every virtual host. It seems that the problem is here. Because > > if I remove the include of httpd-vhosts.conf no segfaults occur. > > > > here my httpd-vhosts.conf with nss related parameters. Are these > > parameters capable for a virtual host configuration? > > You've included some directives that are expected to be global: > > NSSPassPhraseHelper > NSSPassPhraseDialog > NSSSession*Cache* > NSSCertificateDatabase > > I doubt they hurt anything. ok thx, I removed these directives from the httpd-vhosts.conf > > I still can't duplicate a crash. I ran it through valgrind to see if > there was a memory issue and came up with nothing too. > > This could be an architecture problem that I just can't see on x86 > hardware I suppose. I'am using a arm hardware (imx6) > This particular crash seems rather strange too. Within mod_nss enabling > this just calls: > > SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, > mctx->sc->session_tickets); > > Basically enabling it in the model server socket. I would imagine the > crash is probably deeper in NSS. > To narrow things down you might try the NSS tool selfserv. It has an > option, -u, to enable session tickets. It might eliminate mod_nss as the > crash source (or it might implicate it too). thx for the hint to the NSS selfserv tool. I'am using it with DSA Keys instead of Eliptic Curve Keys because selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) First I generate a new nss db. certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf Then starting selfserv with params: selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u selfserv: About to call accept. Now I perform a https request with the Chrome Browser GET /test.php HTTP/1.1 Host: 192.168.1.229 Connection: keep-alive Cache-Control: max-age=0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 DNT: 1 Accept-Encoding: gzip, deflate, sdch Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 EOF it looks for me that it works with selfserv. But I'm not sure if the Session Ticket works and if it makes a difference not using ECC Keys. Best Regards, Oliver From oliver.graute at gmail.com Thu Feb 25 15:12:41 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Thu, 25 Feb 2016 16:12:41 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160225141039.GB29619@graute-opti> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> Message-ID: <20160225151241.GC29619@graute-opti> On 25/02/16, Oliver Graute wrote: > On 24/02/16, Rob Crittenden wrote: > > Oliver Graute wrote: > > > On 23/02/16, Oliver Graute wrote: > > >> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: > > >>> On 23/02/16, Rob Crittenden wrote: > > >>>> Oliver Graute wrote: > > >>>>> On 22/02/16, Rob Crittenden wrote: > > >>>>>> Oliver Graute wrote: > > >>>>>>> Hello, > > >>>>>>> > > >>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > > >>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket > > >>>>>>> feature in my nss.conf with: > > >>>>>>> > > >>>>>>> #RFC 5077 > > >>>>>>> NSSSessionTickets on > > >>>>>>> > > >>>>>>> then something is broken, I see segfaults in my apache error log: > > >>>>>>> > > >>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > > >>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > > >>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > > >>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > > >>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > > >>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > > >>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > > >>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > > >>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > > >>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > > >>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > > >>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > > >>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > > >>>>>>> > > >>>>>>> > > >>>>>>> and in Chrome Browser I got: > > >>>>>>> > > >>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH > > >>>>>>> > > >>>>>>> I tested also a basic ssl client connection with openssl: > > >>>>>>> > > >>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug > > >>>>>>> > > >>>>>>> SSL_connect:SSLv3 read server certificate A > > >>>>>>> SSL_connect:SSLv3 read server key exchange A > > >>>>>>> SSL_connect:SSLv3 read server done A > > >>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > > >>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > > >>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > > >>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > > >>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > > >>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > > >>>>>>> SSL_connect:SSLv3 write client key exchange A > > >>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > > >>>>>>> 0000 - 14 03 03 00 01 01 ...... > > >>>>>>> SSL_connect:SSLv3 write change cipher spec A > > >>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > > >>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > > >>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > > >>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > > >>>>>>> SSL_connect:SSLv3 write finished A > > >>>>>>> SSL_connect:SSLv3 flush data > > >>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > > >>>>>>> SSL_connect:failed in SSLv3 read server session ticket A > > >>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > > >>>>>>> > > >>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. > > >>>>>>> > > >>>>>>> some ideas, whats going on here? > > >>>>>> > > >>>>>> Can you get a stack trace from the core? > > >>>>> > > >>>>> I can give you an strace, see below. Other stack tools are currently not > > >>>>> available, because I need to compile them first for my yocto > > >>>>> environment. If you need something special please tell me. > > >>>>> > > >>>>>> This is Apache 2.4.x? > > >>>>> > > >>>>> yes it is Apache 2.4.16 > > >>>>> > > >>>>>> Is it failing on a request or on startup? > > >>>>> > > >>>>> its failing on every https request. > > >>>> > > >>>> strace in this case isn't particularly helpful as it doesn't show where > > >>>> it is crashing. > > >>>> > > >>>> Can I see your nss.conf? > > >>>> > > >>>> What version of NSS are you using? > > >>> > > >>> I'am using nss in version 3.19.2 > > >>> > > >>> here my nss.conf > > >>> > > >>> # > > >>> # This is the Apache server configuration file providing SSL support using. > > >>> # the mod_nss plugin. It contains the configuration directives to instruct > > >>> # the server how to serve pages over an https connection. > > >>> # > > >>> # Do NOT simply read the instructions in here without understanding > > >>> # what they do. They're here only as hints or reminders. If you are unsure > > >>> # consult the online docs. You have been warned. > > >>> # > > >>> > > >>> # > > >>> # When we also provide SSL we have to listen to the > > >>> # standard HTTP port (see above) and to the HTTPS port > > >>> # > > >>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two > > >>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" > > >>> # > > >>> Listen 443 > > >>> > > >>> ## > > >>> ## SSL Global Context > > >>> ## > > >>> ## All SSL configuration in this context applies both to > > >>> ## the main server and all SSL-enabled virtual hosts. > > >>> ## > > >>> > > >>> # > > >>> # Some MIME-types for downloading Certificates and CRLs > > >>> # > > >>> AddType application/x-x509-ca-cert .crt > > >>> AddType application/x-pkcs7-crl .crl > > >>> > > >>> > > >>> # Pass Phrase Helper: > > >>> # This helper program stores the token password pins between > > >>> # restarts of Apache. > > >>> # Unfortunately the directive is required even if we use no password > > >>> # (though in such case the program should never be used) > > >>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache > > >>> > > >>> # Pass Phrase Dialog: > > >>> # Configure the pass phrase gathering process. > > >>> # The filtering dialog program (`builtin' is a internal > > >>> # terminal dialog) has to provide the pass phrase on stdout. > > >>> #NSSPassPhraseDialog builtin > > >>> NSSPassPhraseDialog file:/etc/apache2/password.conf > > >>> > > >>> # Configure the SSL Session Cache. > > >>> # NSSSessionCacheSize is the number of entries in the cache. > > >>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). > > >>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). > > >>> NSSSessionCacheSize 10000 > > >>> NSSSessionCacheTimeout 100 > > >>> NSSSession3CacheTimeout 86400 > > >>> > > >>> #RFC 5077 > > >>> NSSSessionTickets off > > >> > > >> SessionTickets are on not off > > >> > > >> NSSSessionTickets on > > >> > > > > > > in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss > > > parameters for every virtual host. It seems that the problem is here. Because > > > if I remove the include of httpd-vhosts.conf no segfaults occur. > > > > > > here my httpd-vhosts.conf with nss related parameters. Are these > > > parameters capable for a virtual host configuration? > > > > You've included some directives that are expected to be global: > > > > NSSPassPhraseHelper > > NSSPassPhraseDialog > > NSSSession*Cache* > > NSSCertificateDatabase > > > > I doubt they hurt anything. > > ok thx, I removed these directives from the httpd-vhosts.conf > > > > > I still can't duplicate a crash. I ran it through valgrind to see if > > there was a memory issue and came up with nothing too. > > > > This could be an architecture problem that I just can't see on x86 > > hardware I suppose. > > I'am using a arm hardware (imx6) > > > This particular crash seems rather strange too. Within mod_nss enabling > > this just calls: > > > > SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, > > mctx->sc->session_tickets); > > > > Basically enabling it in the model server socket. I would imagine the > > crash is probably deeper in NSS. > > > > To narrow things down you might try the NSS tool selfserv. It has an > > option, -u, to enable session tickets. It might eliminate mod_nss as the > > crash source (or it might implicate it too). > > > thx for the hint to the NSS selfserv tool. > > I'am using it with DSA Keys instead of Eliptic Curve Keys because > selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) > > First I generate a new nss db. > > certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf > > Then starting selfserv with params: > > selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u > > selfserv: About to call accept. > > Now I perform a https request with the Chrome Browser > > GET /test.php HTTP/1.1 > Host: 192.168.1.229 > Connection: keep-alive > Cache-Control: max-age=0 > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 > Upgrade-Insecure-Requests: 1 > User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 > DNT: 1 > Accept-Encoding: gzip, deflate, sdch > Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 > > EOF > > it looks for me that it works with selfserv. But I'm not sure if the Session > Ticket works and if it makes a difference not using ECC Keys. I checked with wireshark, and I saw the TLS Session Ticket Extension in the hello packet there. I'also switched back to my old ciphers in my apache setup nss.conf NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha and this is working together with NSSSessionTickets on. So it breaks if i'am using ECC Ciphersuite together with Session Tickets! NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256 NSSSessionTickets on is this something that can be explained? Best Regards, Oliver From rcritten at redhat.com Thu Feb 25 15:33:32 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 25 Feb 2016 10:33:32 -0500 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160225151241.GC29619@graute-opti> References: <20160222125434.GA24698@graute-opti> <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> Message-ID: <56CF1ECC.4080303@redhat.com> Oliver Graute wrote: > On 25/02/16, Oliver Graute wrote: >> On 24/02/16, Rob Crittenden wrote: >>> Oliver Graute wrote: >>>> On 23/02/16, Oliver Graute wrote: >>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: >>>>>> On 23/02/16, Rob Crittenden wrote: >>>>>>> Oliver Graute wrote: >>>>>>>> On 22/02/16, Rob Crittenden wrote: >>>>>>>>> Oliver Graute wrote: >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, >>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket >>>>>>>>>> feature in my nss.conf with: >>>>>>>>>> >>>>>>>>>> #RFC 5077 >>>>>>>>>> NSSSessionTickets on >>>>>>>>>> >>>>>>>>>> then something is broken, I see segfaults in my apache error log: >>>>>>>>>> >>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations >>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 >>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' >>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) >>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled >>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) >>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled >>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) >>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled >>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> and in Chrome Browser I got: >>>>>>>>>> >>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH >>>>>>>>>> >>>>>>>>>> I tested also a basic ssl client connection with openssl: >>>>>>>>>> >>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug >>>>>>>>>> >>>>>>>>>> SSL_connect:SSLv3 read server certificate A >>>>>>>>>> SSL_connect:SSLv3 read server key exchange A >>>>>>>>>> SSL_connect:SSLv3 read server done A >>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) >>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE >>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... >>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... >>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... >>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... >>>>>>>>>> SSL_connect:SSLv3 write client key exchange A >>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) >>>>>>>>>> 0000 - 14 03 03 00 01 01 ...... >>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A >>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) >>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. >>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ >>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' >>>>>>>>>> SSL_connect:SSLv3 write finished A >>>>>>>>>> SSL_connect:SSLv3 flush data >>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) >>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A >>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: >>>>>>>>>> >>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. >>>>>>>>>> >>>>>>>>>> some ideas, whats going on here? >>>>>>>>> >>>>>>>>> Can you get a stack trace from the core? >>>>>>>> >>>>>>>> I can give you an strace, see below. Other stack tools are currently not >>>>>>>> available, because I need to compile them first for my yocto >>>>>>>> environment. If you need something special please tell me. >>>>>>>> >>>>>>>>> This is Apache 2.4.x? >>>>>>>> >>>>>>>> yes it is Apache 2.4.16 >>>>>>>> >>>>>>>>> Is it failing on a request or on startup? >>>>>>>> >>>>>>>> its failing on every https request. >>>>>>> >>>>>>> strace in this case isn't particularly helpful as it doesn't show where >>>>>>> it is crashing. >>>>>>> >>>>>>> Can I see your nss.conf? >>>>>>> >>>>>>> What version of NSS are you using? >>>>>> >>>>>> I'am using nss in version 3.19.2 >>>>>> >>>>>> here my nss.conf >>>>>> >>>>>> # >>>>>> # This is the Apache server configuration file providing SSL support using. >>>>>> # the mod_nss plugin. It contains the configuration directives to instruct >>>>>> # the server how to serve pages over an https connection. >>>>>> # >>>>>> # Do NOT simply read the instructions in here without understanding >>>>>> # what they do. They're here only as hints or reminders. If you are unsure >>>>>> # consult the online docs. You have been warned. >>>>>> # >>>>>> >>>>>> # >>>>>> # When we also provide SSL we have to listen to the >>>>>> # standard HTTP port (see above) and to the HTTPS port >>>>>> # >>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two >>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" >>>>>> # >>>>>> Listen 443 >>>>>> >>>>>> ## >>>>>> ## SSL Global Context >>>>>> ## >>>>>> ## All SSL configuration in this context applies both to >>>>>> ## the main server and all SSL-enabled virtual hosts. >>>>>> ## >>>>>> >>>>>> # >>>>>> # Some MIME-types for downloading Certificates and CRLs >>>>>> # >>>>>> AddType application/x-x509-ca-cert .crt >>>>>> AddType application/x-pkcs7-crl .crl >>>>>> >>>>>> >>>>>> # Pass Phrase Helper: >>>>>> # This helper program stores the token password pins between >>>>>> # restarts of Apache. >>>>>> # Unfortunately the directive is required even if we use no password >>>>>> # (though in such case the program should never be used) >>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache >>>>>> >>>>>> # Pass Phrase Dialog: >>>>>> # Configure the pass phrase gathering process. >>>>>> # The filtering dialog program (`builtin' is a internal >>>>>> # terminal dialog) has to provide the pass phrase on stdout. >>>>>> #NSSPassPhraseDialog builtin >>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf >>>>>> >>>>>> # Configure the SSL Session Cache. >>>>>> # NSSSessionCacheSize is the number of entries in the cache. >>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). >>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). >>>>>> NSSSessionCacheSize 10000 >>>>>> NSSSessionCacheTimeout 100 >>>>>> NSSSession3CacheTimeout 86400 >>>>>> >>>>>> #RFC 5077 >>>>>> NSSSessionTickets off >>>>> >>>>> SessionTickets are on not off >>>>> >>>>> NSSSessionTickets on >>>>> >>>> >>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss >>>> parameters for every virtual host. It seems that the problem is here. Because >>>> if I remove the include of httpd-vhosts.conf no segfaults occur. >>>> >>>> here my httpd-vhosts.conf with nss related parameters. Are these >>>> parameters capable for a virtual host configuration? >>> >>> You've included some directives that are expected to be global: >>> >>> NSSPassPhraseHelper >>> NSSPassPhraseDialog >>> NSSSession*Cache* >>> NSSCertificateDatabase >>> >>> I doubt they hurt anything. >> >> ok thx, I removed these directives from the httpd-vhosts.conf >> >>> >>> I still can't duplicate a crash. I ran it through valgrind to see if >>> there was a memory issue and came up with nothing too. >>> >>> This could be an architecture problem that I just can't see on x86 >>> hardware I suppose. >> >> I'am using a arm hardware (imx6) >> >>> This particular crash seems rather strange too. Within mod_nss enabling >>> this just calls: >>> >>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, >>> mctx->sc->session_tickets); >>> >>> Basically enabling it in the model server socket. I would imagine the >>> crash is probably deeper in NSS. >> >> >>> To narrow things down you might try the NSS tool selfserv. It has an >>> option, -u, to enable session tickets. It might eliminate mod_nss as the >>> crash source (or it might implicate it too). >> >> >> thx for the hint to the NSS selfserv tool. >> >> I'am using it with DSA Keys instead of Eliptic Curve Keys because >> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) >> >> First I generate a new nss db. >> >> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf >> >> Then starting selfserv with params: >> >> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u >> >> selfserv: About to call accept. >> >> Now I perform a https request with the Chrome Browser >> >> GET /test.php HTTP/1.1 >> Host: 192.168.1.229 >> Connection: keep-alive >> Cache-Control: max-age=0 >> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 >> Upgrade-Insecure-Requests: 1 >> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 >> DNT: 1 >> Accept-Encoding: gzip, deflate, sdch >> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 >> >> EOF >> >> it looks for me that it works with selfserv. But I'm not sure if the Session >> Ticket works and if it makes a difference not using ECC Keys. > > I checked with wireshark, and I saw the TLS Session Ticket Extension > in the hello packet there. > > I'also switched back to my old ciphers in my apache setup nss.conf > > NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha > > and this is working together with NSSSessionTickets on. > > So it breaks if i'am using ECC Ciphersuite together with Session Tickets! > > NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256 > NSSSessionTickets on > > is this something that can be explained? Hmm, interesting. It gives me something to look at. I was in the middle of responding to your previous message when this one came in. I had been testing with an RSA key. Let me try an ECC key to see if I can reproduce the crash (no success yet). Here is the previous bit I was going to send: This actually generated an RSA key for the cert you used for selfserv. certutil will generate a certificate if the -k option isn't provided. You can do the same with ECC by adding to the last command -k ec -q rob From oliver.graute at gmail.com Fri Feb 26 13:19:12 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Fri, 26 Feb 2016 14:19:12 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56CF1ECC.4080303@redhat.com> References: <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> Message-ID: <20160226131912.GD29619@graute-opti> On 25/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > On 25/02/16, Oliver Graute wrote: > >> On 24/02/16, Rob Crittenden wrote: > >>> Oliver Graute wrote: > >>>> On 23/02/16, Oliver Graute wrote: > >>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: > >>>>>> On 23/02/16, Rob Crittenden wrote: > >>>>>>> Oliver Graute wrote: > >>>>>>>> On 22/02/16, Rob Crittenden wrote: > >>>>>>>>> Oliver Graute wrote: > >>>>>>>>>> Hello, > >>>>>>>>>> > >>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > >>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket > >>>>>>>>>> feature in my nss.conf with: > >>>>>>>>>> > >>>>>>>>>> #RFC 5077 > >>>>>>>>>> NSSSessionTickets on > >>>>>>>>>> > >>>>>>>>>> then something is broken, I see segfaults in my apache error log: > >>>>>>>>>> > >>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > >>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > >>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > >>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > >>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > >>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > >>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> and in Chrome Browser I got: > >>>>>>>>>> > >>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH > >>>>>>>>>> > >>>>>>>>>> I tested also a basic ssl client connection with openssl: > >>>>>>>>>> > >>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug > >>>>>>>>>> > >>>>>>>>>> SSL_connect:SSLv3 read server certificate A > >>>>>>>>>> SSL_connect:SSLv3 read server key exchange A > >>>>>>>>>> SSL_connect:SSLv3 read server done A > >>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > >>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > >>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > >>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > >>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > >>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > >>>>>>>>>> SSL_connect:SSLv3 write client key exchange A > >>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > >>>>>>>>>> 0000 - 14 03 03 00 01 01 ...... > >>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A > >>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > >>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > >>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > >>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > >>>>>>>>>> SSL_connect:SSLv3 write finished A > >>>>>>>>>> SSL_connect:SSLv3 flush data > >>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > >>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A > >>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > >>>>>>>>>> > >>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. > >>>>>>>>>> > >>>>>>>>>> some ideas, whats going on here? > >>>>>>>>> > >>>>>>>>> Can you get a stack trace from the core? > >>>>>>>> > >>>>>>>> I can give you an strace, see below. Other stack tools are currently not > >>>>>>>> available, because I need to compile them first for my yocto > >>>>>>>> environment. If you need something special please tell me. > >>>>>>>> > >>>>>>>>> This is Apache 2.4.x? > >>>>>>>> > >>>>>>>> yes it is Apache 2.4.16 > >>>>>>>> > >>>>>>>>> Is it failing on a request or on startup? > >>>>>>>> > >>>>>>>> its failing on every https request. > >>>>>>> > >>>>>>> strace in this case isn't particularly helpful as it doesn't show where > >>>>>>> it is crashing. > >>>>>>> > >>>>>>> Can I see your nss.conf? > >>>>>>> > >>>>>>> What version of NSS are you using? > >>>>>> > >>>>>> I'am using nss in version 3.19.2 > >>>>>> > >>>>>> here my nss.conf > >>>>>> > >>>>>> # > >>>>>> # This is the Apache server configuration file providing SSL support using. > >>>>>> # the mod_nss plugin. It contains the configuration directives to instruct > >>>>>> # the server how to serve pages over an https connection. > >>>>>> # > >>>>>> # Do NOT simply read the instructions in here without understanding > >>>>>> # what they do. They're here only as hints or reminders. If you are unsure > >>>>>> # consult the online docs. You have been warned. > >>>>>> # > >>>>>> > >>>>>> # > >>>>>> # When we also provide SSL we have to listen to the > >>>>>> # standard HTTP port (see above) and to the HTTPS port > >>>>>> # > >>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two > >>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" > >>>>>> # > >>>>>> Listen 443 > >>>>>> > >>>>>> ## > >>>>>> ## SSL Global Context > >>>>>> ## > >>>>>> ## All SSL configuration in this context applies both to > >>>>>> ## the main server and all SSL-enabled virtual hosts. > >>>>>> ## > >>>>>> > >>>>>> # > >>>>>> # Some MIME-types for downloading Certificates and CRLs > >>>>>> # > >>>>>> AddType application/x-x509-ca-cert .crt > >>>>>> AddType application/x-pkcs7-crl .crl > >>>>>> > >>>>>> > >>>>>> # Pass Phrase Helper: > >>>>>> # This helper program stores the token password pins between > >>>>>> # restarts of Apache. > >>>>>> # Unfortunately the directive is required even if we use no password > >>>>>> # (though in such case the program should never be used) > >>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache > >>>>>> > >>>>>> # Pass Phrase Dialog: > >>>>>> # Configure the pass phrase gathering process. > >>>>>> # The filtering dialog program (`builtin' is a internal > >>>>>> # terminal dialog) has to provide the pass phrase on stdout. > >>>>>> #NSSPassPhraseDialog builtin > >>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf > >>>>>> > >>>>>> # Configure the SSL Session Cache. > >>>>>> # NSSSessionCacheSize is the number of entries in the cache. > >>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). > >>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). > >>>>>> NSSSessionCacheSize 10000 > >>>>>> NSSSessionCacheTimeout 100 > >>>>>> NSSSession3CacheTimeout 86400 > >>>>>> > >>>>>> #RFC 5077 > >>>>>> NSSSessionTickets off > >>>>> > >>>>> SessionTickets are on not off > >>>>> > >>>>> NSSSessionTickets on > >>>>> > >>>> > >>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss > >>>> parameters for every virtual host. It seems that the problem is here. Because > >>>> if I remove the include of httpd-vhosts.conf no segfaults occur. > >>>> > >>>> here my httpd-vhosts.conf with nss related parameters. Are these > >>>> parameters capable for a virtual host configuration? > >>> > >>> You've included some directives that are expected to be global: > >>> > >>> NSSPassPhraseHelper > >>> NSSPassPhraseDialog > >>> NSSSession*Cache* > >>> NSSCertificateDatabase > >>> > >>> I doubt they hurt anything. > >> > >> ok thx, I removed these directives from the httpd-vhosts.conf > >> > >>> > >>> I still can't duplicate a crash. I ran it through valgrind to see if > >>> there was a memory issue and came up with nothing too. > >>> > >>> This could be an architecture problem that I just can't see on x86 > >>> hardware I suppose. > >> > >> I'am using a arm hardware (imx6) > >> > >>> This particular crash seems rather strange too. Within mod_nss enabling > >>> this just calls: > >>> > >>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, > >>> mctx->sc->session_tickets); > >>> > >>> Basically enabling it in the model server socket. I would imagine the > >>> crash is probably deeper in NSS. > >> > >> > >>> To narrow things down you might try the NSS tool selfserv. It has an > >>> option, -u, to enable session tickets. It might eliminate mod_nss as the > >>> crash source (or it might implicate it too). > >> > >> > >> thx for the hint to the NSS selfserv tool. > >> > >> I'am using it with DSA Keys instead of Eliptic Curve Keys because > >> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) > >> > >> First I generate a new nss db. > >> > >> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf > >> > >> Then starting selfserv with params: > >> > >> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u > >> > >> selfserv: About to call accept. > >> > >> Now I perform a https request with the Chrome Browser > >> > >> GET /test.php HTTP/1.1 > >> Host: 192.168.1.229 > >> Connection: keep-alive > >> Cache-Control: max-age=0 > >> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 > >> Upgrade-Insecure-Requests: 1 > >> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 > >> DNT: 1 > >> Accept-Encoding: gzip, deflate, sdch > >> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 > >> > >> EOF > >> > >> it looks for me that it works with selfserv. But I'm not sure if the Session > >> Ticket works and if it makes a difference not using ECC Keys. > > > > I checked with wireshark, and I saw the TLS Session Ticket Extension > > in the hello packet there. > > > > I'also switched back to my old ciphers in my apache setup nss.conf > > > > NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha > > > > and this is working together with NSSSessionTickets on. > > > > So it breaks if i'am using ECC Ciphersuite together with Session Tickets! > > > > NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256 > > NSSSessionTickets on > > > > is this something that can be explained? > > Hmm, interesting. It gives me something to look at. I was in the middle > of responding to your previous message when this one came in. > > I had been testing with an RSA key. Let me try an ECC key to see if I > can reproduce the crash (no success yet). > > Here is the previous bit I was going to send: > > This actually generated an RSA key for the cert you used for selfserv. > certutil will generate a certificate if the -k option isn't provided. > > You can do the same with ECC by adding to the last command -k ec -q I updated from nss 3.19 to nss 3.21 but I run into the same issue with it. Then I played around with gdb and saw that it breaks in libssl3.so Some hints how I can debug this further? Best Regards, Oliver gdb httpd run -X -e debug -D /usr/sbin/httpd root at ibis-box:/var/apache2/logs# gdb httpd GNU gdb (GDB) 7.9.1 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-poky-linux-gnueabi". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from httpd...(no debugging symbols found)...done. (gdb) run -X -e debug -D /usr/sbin/httpd Starting program: /usr/sbin/httpd -X -e debug -D /usr/sbin/httpd warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". To enable execution of this file add add-auto-load-safe-path /lib/libthread_db-1.0.so line to your configuration file "/home/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/home/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. [Fri Feb 26 00:03:18.648483 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so [Fri Feb 26 00:03:18.700565 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so [Fri Feb 26 00:03:18.750258 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so [Fri Feb 26 00:03:18.801825 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so [Fri Feb 26 00:03:18.852520 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so [Fri Feb 26 00:03:18.909582 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so [Fri Feb 26 00:03:18.964388 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so [Fri Feb 26 00:03:19.022456 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so [Fri Feb 26 00:03:19.081787 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so [Fri Feb 26 00:03:19.142737 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so [Fri Feb 26 00:03:19.205120 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so [Fri Feb 26 00:03:19.288557 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so [Fri Feb 26 00:03:19.355114 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so [Fri Feb 26 00:03:19.428930 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module log_config_module from /usr/lib/apache2/modules/mod_log_config.so [Fri Feb 26 00:03:19.495578 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so [Fri Feb 26 00:03:19.569779 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so [Fri Feb 26 00:03:19.641764 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so [Fri Feb 26 00:03:19.716852 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module version_module from /usr/lib/apache2/modules/mod_version.so [Fri Feb 26 00:03:20.213946 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module nss_module from /usr/lib/apache2/modules/libmodnss.so [Fri Feb 26 00:03:20.307607 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so [Fri Feb 26 00:03:20.393425 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module unixd_module from /usr/lib/apache2/modules/mod_unixd.so [Fri Feb 26 00:03:20.483090 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so [Fri Feb 26 00:03:20.578632 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so [Fri Feb 26 00:03:20.671102 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so [Fri Feb 26 00:03:20.763080 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so Program received signal SIGSEGV, Segmentation fault. 0x76b366dc in ?? () from /usr/lib/libssl3.so From rcritten at redhat.com Fri Feb 26 15:32:57 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 26 Feb 2016 10:32:57 -0500 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160226131912.GD29619@graute-opti> References: <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> <20160226131912.GD29619@graute-opti> Message-ID: <56D07029.7050202@redhat.com> Oliver Graute wrote: > On 25/02/16, Rob Crittenden wrote: >> Oliver Graute wrote: >>> On 25/02/16, Oliver Graute wrote: >>>> On 24/02/16, Rob Crittenden wrote: >>>>> Oliver Graute wrote: >>>>>> On 23/02/16, Oliver Graute wrote: >>>>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: >>>>>>>> On 23/02/16, Rob Crittenden wrote: >>>>>>>>> Oliver Graute wrote: >>>>>>>>>> On 22/02/16, Rob Crittenden wrote: >>>>>>>>>>> Oliver Graute wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, >>>>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket >>>>>>>>>>>> feature in my nss.conf with: >>>>>>>>>>>> >>>>>>>>>>>> #RFC 5077 >>>>>>>>>>>> NSSSessionTickets on >>>>>>>>>>>> >>>>>>>>>>>> then something is broken, I see segfaults in my apache error log: >>>>>>>>>>>> >>>>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations >>>>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 >>>>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' >>>>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) >>>>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) >>>>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) >>>>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> and in Chrome Browser I got: >>>>>>>>>>>> >>>>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH >>>>>>>>>>>> >>>>>>>>>>>> I tested also a basic ssl client connection with openssl: >>>>>>>>>>>> >>>>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug >>>>>>>>>>>> >>>>>>>>>>>> SSL_connect:SSLv3 read server certificate A >>>>>>>>>>>> SSL_connect:SSLv3 read server key exchange A >>>>>>>>>>>> SSL_connect:SSLv3 read server done A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) >>>>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE >>>>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... >>>>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... >>>>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... >>>>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... >>>>>>>>>>>> SSL_connect:SSLv3 write client key exchange A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) >>>>>>>>>>>> 0000 - 14 03 03 00 01 01 ...... >>>>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) >>>>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. >>>>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ >>>>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' >>>>>>>>>>>> SSL_connect:SSLv3 write finished A >>>>>>>>>>>> SSL_connect:SSLv3 flush data >>>>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) >>>>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A >>>>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: >>>>>>>>>>>> >>>>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. >>>>>>>>>>>> >>>>>>>>>>>> some ideas, whats going on here? >>>>>>>>>>> >>>>>>>>>>> Can you get a stack trace from the core? >>>>>>>>>> >>>>>>>>>> I can give you an strace, see below. Other stack tools are currently not >>>>>>>>>> available, because I need to compile them first for my yocto >>>>>>>>>> environment. If you need something special please tell me. >>>>>>>>>> >>>>>>>>>>> This is Apache 2.4.x? >>>>>>>>>> >>>>>>>>>> yes it is Apache 2.4.16 >>>>>>>>>> >>>>>>>>>>> Is it failing on a request or on startup? >>>>>>>>>> >>>>>>>>>> its failing on every https request. >>>>>>>>> >>>>>>>>> strace in this case isn't particularly helpful as it doesn't show where >>>>>>>>> it is crashing. >>>>>>>>> >>>>>>>>> Can I see your nss.conf? >>>>>>>>> >>>>>>>>> What version of NSS are you using? >>>>>>>> >>>>>>>> I'am using nss in version 3.19.2 >>>>>>>> >>>>>>>> here my nss.conf >>>>>>>> >>>>>>>> # >>>>>>>> # This is the Apache server configuration file providing SSL support using. >>>>>>>> # the mod_nss plugin. It contains the configuration directives to instruct >>>>>>>> # the server how to serve pages over an https connection. >>>>>>>> # >>>>>>>> # Do NOT simply read the instructions in here without understanding >>>>>>>> # what they do. They're here only as hints or reminders. If you are unsure >>>>>>>> # consult the online docs. You have been warned. >>>>>>>> # >>>>>>>> >>>>>>>> # >>>>>>>> # When we also provide SSL we have to listen to the >>>>>>>> # standard HTTP port (see above) and to the HTTPS port >>>>>>>> # >>>>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two >>>>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" >>>>>>>> # >>>>>>>> Listen 443 >>>>>>>> >>>>>>>> ## >>>>>>>> ## SSL Global Context >>>>>>>> ## >>>>>>>> ## All SSL configuration in this context applies both to >>>>>>>> ## the main server and all SSL-enabled virtual hosts. >>>>>>>> ## >>>>>>>> >>>>>>>> # >>>>>>>> # Some MIME-types for downloading Certificates and CRLs >>>>>>>> # >>>>>>>> AddType application/x-x509-ca-cert .crt >>>>>>>> AddType application/x-pkcs7-crl .crl >>>>>>>> >>>>>>>> >>>>>>>> # Pass Phrase Helper: >>>>>>>> # This helper program stores the token password pins between >>>>>>>> # restarts of Apache. >>>>>>>> # Unfortunately the directive is required even if we use no password >>>>>>>> # (though in such case the program should never be used) >>>>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache >>>>>>>> >>>>>>>> # Pass Phrase Dialog: >>>>>>>> # Configure the pass phrase gathering process. >>>>>>>> # The filtering dialog program (`builtin' is a internal >>>>>>>> # terminal dialog) has to provide the pass phrase on stdout. >>>>>>>> #NSSPassPhraseDialog builtin >>>>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf >>>>>>>> >>>>>>>> # Configure the SSL Session Cache. >>>>>>>> # NSSSessionCacheSize is the number of entries in the cache. >>>>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). >>>>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). >>>>>>>> NSSSessionCacheSize 10000 >>>>>>>> NSSSessionCacheTimeout 100 >>>>>>>> NSSSession3CacheTimeout 86400 >>>>>>>> >>>>>>>> #RFC 5077 >>>>>>>> NSSSessionTickets off >>>>>>> >>>>>>> SessionTickets are on not off >>>>>>> >>>>>>> NSSSessionTickets on >>>>>>> >>>>>> >>>>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss >>>>>> parameters for every virtual host. It seems that the problem is here. Because >>>>>> if I remove the include of httpd-vhosts.conf no segfaults occur. >>>>>> >>>>>> here my httpd-vhosts.conf with nss related parameters. Are these >>>>>> parameters capable for a virtual host configuration? >>>>> >>>>> You've included some directives that are expected to be global: >>>>> >>>>> NSSPassPhraseHelper >>>>> NSSPassPhraseDialog >>>>> NSSSession*Cache* >>>>> NSSCertificateDatabase >>>>> >>>>> I doubt they hurt anything. >>>> >>>> ok thx, I removed these directives from the httpd-vhosts.conf >>>> >>>>> >>>>> I still can't duplicate a crash. I ran it through valgrind to see if >>>>> there was a memory issue and came up with nothing too. >>>>> >>>>> This could be an architecture problem that I just can't see on x86 >>>>> hardware I suppose. >>>> >>>> I'am using a arm hardware (imx6) >>>> >>>>> This particular crash seems rather strange too. Within mod_nss enabling >>>>> this just calls: >>>>> >>>>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, >>>>> mctx->sc->session_tickets); >>>>> >>>>> Basically enabling it in the model server socket. I would imagine the >>>>> crash is probably deeper in NSS. >>>> >>>> >>>>> To narrow things down you might try the NSS tool selfserv. It has an >>>>> option, -u, to enable session tickets. It might eliminate mod_nss as the >>>>> crash source (or it might implicate it too). >>>> >>>> >>>> thx for the hint to the NSS selfserv tool. >>>> >>>> I'am using it with DSA Keys instead of Eliptic Curve Keys because >>>> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) >>>> >>>> First I generate a new nss db. >>>> >>>> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf >>>> >>>> Then starting selfserv with params: >>>> >>>> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u >>>> >>>> selfserv: About to call accept. >>>> >>>> Now I perform a https request with the Chrome Browser >>>> >>>> GET /test.php HTTP/1.1 >>>> Host: 192.168.1.229 >>>> Connection: keep-alive >>>> Cache-Control: max-age=0 >>>> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 >>>> Upgrade-Insecure-Requests: 1 >>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 >>>> DNT: 1 >>>> Accept-Encoding: gzip, deflate, sdch >>>> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 >>>> >>>> EOF >>>> >>>> it looks for me that it works with selfserv. But I'm not sure if the Session >>>> Ticket works and if it makes a difference not using ECC Keys. >>> >>> I checked with wireshark, and I saw the TLS Session Ticket Extension >>> in the hello packet there. >>> >>> I'also switched back to my old ciphers in my apache setup nss.conf >>> >>> NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha >>> >>> and this is working together with NSSSessionTickets on. >>> >>> So it breaks if i'am using ECC Ciphersuite together with Session Tickets! >>> >>> NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256 >>> NSSSessionTickets on >>> >>> is this something that can be explained? >> >> Hmm, interesting. It gives me something to look at. I was in the middle >> of responding to your previous message when this one came in. >> >> I had been testing with an RSA key. Let me try an ECC key to see if I >> can reproduce the crash (no success yet). >> >> Here is the previous bit I was going to send: >> >> This actually generated an RSA key for the cert you used for selfserv. >> certutil will generate a certificate if the -k option isn't provided. >> >> You can do the same with ECC by adding to the last command -k ec -q > > > I updated from nss 3.19 to nss 3.21 but I run into the same issue with it. > > Then I played around with gdb and saw that it breaks in libssl3.so > Some hints how I can debug this further? > > Best Regards, > > Oliver > > gdb httpd > run -X -e debug -D /usr/sbin/httpd > > root at ibis-box:/var/apache2/logs# gdb httpd > GNU gdb (GDB) 7.9.1 > Copyright (C) 2015 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "arm-poky-linux-gnueabi". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: > . > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from httpd...(no debugging symbols found)...done. > (gdb) run -X -e debug -D /usr/sbin/httpd > Starting program: /usr/sbin/httpd -X -e debug -D /usr/sbin/httpd > warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". > To enable execution of this file add > add-auto-load-safe-path /lib/libthread_db-1.0.so > line to your configuration file "/home/root/.gdbinit". > To completely disable this security protection add > set auto-load safe-path / > line to your configuration file "/home/root/.gdbinit". > For more information about this security protection see the > "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: > info "(gdb)Auto-loading safe path" > warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. > [Fri Feb 26 00:03:18.648483 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so > [Fri Feb 26 00:03:18.700565 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so > [Fri Feb 26 00:03:18.750258 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so > [Fri Feb 26 00:03:18.801825 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so > [Fri Feb 26 00:03:18.852520 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so > [Fri Feb 26 00:03:18.909582 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so > [Fri Feb 26 00:03:18.964388 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so > [Fri Feb 26 00:03:19.022456 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so > [Fri Feb 26 00:03:19.081787 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so > [Fri Feb 26 00:03:19.142737 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so > [Fri Feb 26 00:03:19.205120 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so > [Fri Feb 26 00:03:19.288557 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so > [Fri Feb 26 00:03:19.355114 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so > [Fri Feb 26 00:03:19.428930 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module log_config_module from /usr/lib/apache2/modules/mod_log_config.so > [Fri Feb 26 00:03:19.495578 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so > [Fri Feb 26 00:03:19.569779 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so > [Fri Feb 26 00:03:19.641764 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so > [Fri Feb 26 00:03:19.716852 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module version_module from /usr/lib/apache2/modules/mod_version.so > [Fri Feb 26 00:03:20.213946 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module nss_module from /usr/lib/apache2/modules/libmodnss.so > [Fri Feb 26 00:03:20.307607 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so > [Fri Feb 26 00:03:20.393425 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module unixd_module from /usr/lib/apache2/modules/mod_unixd.so > [Fri Feb 26 00:03:20.483090 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so > [Fri Feb 26 00:03:20.578632 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so > [Fri Feb 26 00:03:20.671102 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so > [Fri Feb 26 00:03:20.763080 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so > > Program received signal SIGSEGV, Segmentation fault. > 0x76b366dc in ?? () from /usr/lib/libssl3.so If you could build NSS with debugging we would be able to see where it is failing. I'm not sure where your package is coming from but IIRC to build with debug you can pass in something at build time, like BUILD_OPT=0 I'm trying to dig up an arm7 machine to see if I can reproduce this in Fedora. rob From rcritten at redhat.com Fri Feb 26 19:27:40 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 26 Feb 2016 14:27:40 -0500 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160226131912.GD29619@graute-opti> References: <56CB19D2.5010307@redhat.com> <20160223085826.GA11923@graute-opti> <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> <20160226131912.GD29619@graute-opti> Message-ID: <56D0A72C.2010506@redhat.com> Oliver Graute wrote: > On 25/02/16, Rob Crittenden wrote: >> Oliver Graute wrote: >>> On 25/02/16, Oliver Graute wrote: >>>> On 24/02/16, Rob Crittenden wrote: >>>>> Oliver Graute wrote: >>>>>> On 23/02/16, Oliver Graute wrote: >>>>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: >>>>>>>> On 23/02/16, Rob Crittenden wrote: >>>>>>>>> Oliver Graute wrote: >>>>>>>>>> On 22/02/16, Rob Crittenden wrote: >>>>>>>>>>> Oliver Graute wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, >>>>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket >>>>>>>>>>>> feature in my nss.conf with: >>>>>>>>>>>> >>>>>>>>>>>> #RFC 5077 >>>>>>>>>>>> NSSSessionTickets on >>>>>>>>>>>> >>>>>>>>>>>> then something is broken, I see segfaults in my apache error log: >>>>>>>>>>>> >>>>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations >>>>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 >>>>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' >>>>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) >>>>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) >>>>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) >>>>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled >>>>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy >>>>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> and in Chrome Browser I got: >>>>>>>>>>>> >>>>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH >>>>>>>>>>>> >>>>>>>>>>>> I tested also a basic ssl client connection with openssl: >>>>>>>>>>>> >>>>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug >>>>>>>>>>>> >>>>>>>>>>>> SSL_connect:SSLv3 read server certificate A >>>>>>>>>>>> SSL_connect:SSLv3 read server key exchange A >>>>>>>>>>>> SSL_connect:SSLv3 read server done A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) >>>>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE >>>>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... >>>>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... >>>>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... >>>>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... >>>>>>>>>>>> SSL_connect:SSLv3 write client key exchange A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) >>>>>>>>>>>> 0000 - 14 03 03 00 01 01 ...... >>>>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) >>>>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. >>>>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ >>>>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' >>>>>>>>>>>> SSL_connect:SSLv3 write finished A >>>>>>>>>>>> SSL_connect:SSLv3 flush data >>>>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) >>>>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A >>>>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: >>>>>>>>>>>> >>>>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. >>>>>>>>>>>> >>>>>>>>>>>> some ideas, whats going on here? >>>>>>>>>>> >>>>>>>>>>> Can you get a stack trace from the core? >>>>>>>>>> >>>>>>>>>> I can give you an strace, see below. Other stack tools are currently not >>>>>>>>>> available, because I need to compile them first for my yocto >>>>>>>>>> environment. If you need something special please tell me. >>>>>>>>>> >>>>>>>>>>> This is Apache 2.4.x? >>>>>>>>>> >>>>>>>>>> yes it is Apache 2.4.16 >>>>>>>>>> >>>>>>>>>>> Is it failing on a request or on startup? >>>>>>>>>> >>>>>>>>>> its failing on every https request. >>>>>>>>> >>>>>>>>> strace in this case isn't particularly helpful as it doesn't show where >>>>>>>>> it is crashing. >>>>>>>>> >>>>>>>>> Can I see your nss.conf? >>>>>>>>> >>>>>>>>> What version of NSS are you using? >>>>>>>> >>>>>>>> I'am using nss in version 3.19.2 >>>>>>>> >>>>>>>> here my nss.conf >>>>>>>> >>>>>>>> # >>>>>>>> # This is the Apache server configuration file providing SSL support using. >>>>>>>> # the mod_nss plugin. It contains the configuration directives to instruct >>>>>>>> # the server how to serve pages over an https connection. >>>>>>>> # >>>>>>>> # Do NOT simply read the instructions in here without understanding >>>>>>>> # what they do. They're here only as hints or reminders. If you are unsure >>>>>>>> # consult the online docs. You have been warned. >>>>>>>> # >>>>>>>> >>>>>>>> # >>>>>>>> # When we also provide SSL we have to listen to the >>>>>>>> # standard HTTP port (see above) and to the HTTPS port >>>>>>>> # >>>>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two >>>>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" >>>>>>>> # >>>>>>>> Listen 443 >>>>>>>> >>>>>>>> ## >>>>>>>> ## SSL Global Context >>>>>>>> ## >>>>>>>> ## All SSL configuration in this context applies both to >>>>>>>> ## the main server and all SSL-enabled virtual hosts. >>>>>>>> ## >>>>>>>> >>>>>>>> # >>>>>>>> # Some MIME-types for downloading Certificates and CRLs >>>>>>>> # >>>>>>>> AddType application/x-x509-ca-cert .crt >>>>>>>> AddType application/x-pkcs7-crl .crl >>>>>>>> >>>>>>>> >>>>>>>> # Pass Phrase Helper: >>>>>>>> # This helper program stores the token password pins between >>>>>>>> # restarts of Apache. >>>>>>>> # Unfortunately the directive is required even if we use no password >>>>>>>> # (though in such case the program should never be used) >>>>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache >>>>>>>> >>>>>>>> # Pass Phrase Dialog: >>>>>>>> # Configure the pass phrase gathering process. >>>>>>>> # The filtering dialog program (`builtin' is a internal >>>>>>>> # terminal dialog) has to provide the pass phrase on stdout. >>>>>>>> #NSSPassPhraseDialog builtin >>>>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf >>>>>>>> >>>>>>>> # Configure the SSL Session Cache. >>>>>>>> # NSSSessionCacheSize is the number of entries in the cache. >>>>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). >>>>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). >>>>>>>> NSSSessionCacheSize 10000 >>>>>>>> NSSSessionCacheTimeout 100 >>>>>>>> NSSSession3CacheTimeout 86400 >>>>>>>> >>>>>>>> #RFC 5077 >>>>>>>> NSSSessionTickets off >>>>>>> >>>>>>> SessionTickets are on not off >>>>>>> >>>>>>> NSSSessionTickets on >>>>>>> >>>>>> >>>>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss >>>>>> parameters for every virtual host. It seems that the problem is here. Because >>>>>> if I remove the include of httpd-vhosts.conf no segfaults occur. >>>>>> >>>>>> here my httpd-vhosts.conf with nss related parameters. Are these >>>>>> parameters capable for a virtual host configuration? >>>>> >>>>> You've included some directives that are expected to be global: >>>>> >>>>> NSSPassPhraseHelper >>>>> NSSPassPhraseDialog >>>>> NSSSession*Cache* >>>>> NSSCertificateDatabase >>>>> >>>>> I doubt they hurt anything. >>>> >>>> ok thx, I removed these directives from the httpd-vhosts.conf >>>> >>>>> >>>>> I still can't duplicate a crash. I ran it through valgrind to see if >>>>> there was a memory issue and came up with nothing too. >>>>> >>>>> This could be an architecture problem that I just can't see on x86 >>>>> hardware I suppose. >>>> >>>> I'am using a arm hardware (imx6) >>>> >>>>> This particular crash seems rather strange too. Within mod_nss enabling >>>>> this just calls: >>>>> >>>>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, >>>>> mctx->sc->session_tickets); >>>>> >>>>> Basically enabling it in the model server socket. I would imagine the >>>>> crash is probably deeper in NSS. >>>> >>>> >>>>> To narrow things down you might try the NSS tool selfserv. It has an >>>>> option, -u, to enable session tickets. It might eliminate mod_nss as the >>>>> crash source (or it might implicate it too). >>>> >>>> >>>> thx for the hint to the NSS selfserv tool. >>>> >>>> I'am using it with DSA Keys instead of Eliptic Curve Keys because >>>> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) >>>> >>>> First I generate a new nss db. >>>> >>>> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf >>>> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf >>>> >>>> Then starting selfserv with params: >>>> >>>> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u >>>> >>>> selfserv: About to call accept. >>>> >>>> Now I perform a https request with the Chrome Browser >>>> >>>> GET /test.php HTTP/1.1 >>>> Host: 192.168.1.229 >>>> Connection: keep-alive >>>> Cache-Control: max-age=0 >>>> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 >>>> Upgrade-Insecure-Requests: 1 >>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 >>>> DNT: 1 >>>> Accept-Encoding: gzip, deflate, sdch >>>> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 >>>> >>>> EOF >>>> >>>> it looks for me that it works with selfserv. But I'm not sure if the Session >>>> Ticket works and if it makes a difference not using ECC Keys. >>> >>> I checked with wireshark, and I saw the TLS Session Ticket Extension >>> in the hello packet there. >>> >>> I'also switched back to my old ciphers in my apache setup nss.conf >>> >>> NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha >>> >>> and this is working together with NSSSessionTickets on. >>> >>> So it breaks if i'am using ECC Ciphersuite together with Session Tickets! >>> >>> NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256 >>> NSSSessionTickets on >>> >>> is this something that can be explained? >> >> Hmm, interesting. It gives me something to look at. I was in the middle >> of responding to your previous message when this one came in. >> >> I had been testing with an RSA key. Let me try an ECC key to see if I >> can reproduce the crash (no success yet). >> >> Here is the previous bit I was going to send: >> >> This actually generated an RSA key for the cert you used for selfserv. >> certutil will generate a certificate if the -k option isn't provided. >> >> You can do the same with ECC by adding to the last command -k ec -q > > > I updated from nss 3.19 to nss 3.21 but I run into the same issue with it. > > Then I played around with gdb and saw that it breaks in libssl3.so > Some hints how I can debug this further? > > Best Regards, > > Oliver > > gdb httpd > run -X -e debug -D /usr/sbin/httpd > > root at ibis-box:/var/apache2/logs# gdb httpd > GNU gdb (GDB) 7.9.1 > Copyright (C) 2015 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "arm-poky-linux-gnueabi". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: > . > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from httpd...(no debugging symbols found)...done. > (gdb) run -X -e debug -D /usr/sbin/httpd > Starting program: /usr/sbin/httpd -X -e debug -D /usr/sbin/httpd > warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". > To enable execution of this file add > add-auto-load-safe-path /lib/libthread_db-1.0.so > line to your configuration file "/home/root/.gdbinit". > To completely disable this security protection add > set auto-load safe-path / > line to your configuration file "/home/root/.gdbinit". > For more information about this security protection see the > "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: > info "(gdb)Auto-loading safe path" > warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. > [Fri Feb 26 00:03:18.648483 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so > [Fri Feb 26 00:03:18.700565 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so > [Fri Feb 26 00:03:18.750258 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so > [Fri Feb 26 00:03:18.801825 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so > [Fri Feb 26 00:03:18.852520 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so > [Fri Feb 26 00:03:18.909582 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so > [Fri Feb 26 00:03:18.964388 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so > [Fri Feb 26 00:03:19.022456 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so > [Fri Feb 26 00:03:19.081787 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so > [Fri Feb 26 00:03:19.142737 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so > [Fri Feb 26 00:03:19.205120 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so > [Fri Feb 26 00:03:19.288557 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so > [Fri Feb 26 00:03:19.355114 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so > [Fri Feb 26 00:03:19.428930 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module log_config_module from /usr/lib/apache2/modules/mod_log_config.so > [Fri Feb 26 00:03:19.495578 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so > [Fri Feb 26 00:03:19.569779 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so > [Fri Feb 26 00:03:19.641764 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so > [Fri Feb 26 00:03:19.716852 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module version_module from /usr/lib/apache2/modules/mod_version.so > [Fri Feb 26 00:03:20.213946 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module nss_module from /usr/lib/apache2/modules/libmodnss.so > [Fri Feb 26 00:03:20.307607 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so > [Fri Feb 26 00:03:20.393425 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module unixd_module from /usr/lib/apache2/modules/mod_unixd.so > [Fri Feb 26 00:03:20.483090 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so > [Fri Feb 26 00:03:20.578632 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so > [Fri Feb 26 00:03:20.671102 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so > [Fri Feb 26 00:03:20.763080 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so > > Program received signal SIGSEGV, Segmentation fault. > 0x76b366dc in ?? () from /usr/lib/libssl3.so Can I ask why you have NSSEnforceValidCerts off set? Is it because of an error that was logged, preventing the server from starting? rob From oliver.graute at gmail.com Mon Feb 29 08:18:17 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Mon, 29 Feb 2016 09:18:17 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56D0A72C.2010506@redhat.com> References: <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> <20160226131912.GD29619@graute-opti> <56D0A72C.2010506@redhat.com> Message-ID: <20160229081817.GA14634@graute-opti> On 26/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > On 25/02/16, Rob Crittenden wrote: > >> Oliver Graute wrote: > >>> On 25/02/16, Oliver Graute wrote: > >>>> On 24/02/16, Rob Crittenden wrote: > >>>>> Oliver Graute wrote: > >>>>>> On 23/02/16, Oliver Graute wrote: > >>>>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: > >>>>>>>> On 23/02/16, Rob Crittenden wrote: > >>>>>>>>> Oliver Graute wrote: > >>>>>>>>>> On 22/02/16, Rob Crittenden wrote: > >>>>>>>>>>> Oliver Graute wrote: > >>>>>>>>>>>> Hello, > >>>>>>>>>>>> > >>>>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > >>>>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket > >>>>>>>>>>>> feature in my nss.conf with: > >>>>>>>>>>>> > >>>>>>>>>>>> #RFC 5077 > >>>>>>>>>>>> NSSSessionTickets on > >>>>>>>>>>>> > >>>>>>>>>>>> then something is broken, I see segfaults in my apache error log: > >>>>>>>>>>>> > >>>>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > >>>>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > >>>>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > >>>>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > >>>>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > >>>>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > >>>>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> and in Chrome Browser I got: > >>>>>>>>>>>> > >>>>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH > >>>>>>>>>>>> > >>>>>>>>>>>> I tested also a basic ssl client connection with openssl: > >>>>>>>>>>>> > >>>>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug > >>>>>>>>>>>> > >>>>>>>>>>>> SSL_connect:SSLv3 read server certificate A > >>>>>>>>>>>> SSL_connect:SSLv3 read server key exchange A > >>>>>>>>>>>> SSL_connect:SSLv3 read server done A > >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > >>>>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > >>>>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > >>>>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > >>>>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > >>>>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > >>>>>>>>>>>> SSL_connect:SSLv3 write client key exchange A > >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > >>>>>>>>>>>> 0000 - 14 03 03 00 01 01 ...... > >>>>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A > >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > >>>>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > >>>>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > >>>>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > >>>>>>>>>>>> SSL_connect:SSLv3 write finished A > >>>>>>>>>>>> SSL_connect:SSLv3 flush data > >>>>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > >>>>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A > >>>>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > >>>>>>>>>>>> > >>>>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. > >>>>>>>>>>>> > >>>>>>>>>>>> some ideas, whats going on here? > >>>>>>>>>>> > >>>>>>>>>>> Can you get a stack trace from the core? > >>>>>>>>>> > >>>>>>>>>> I can give you an strace, see below. Other stack tools are currently not > >>>>>>>>>> available, because I need to compile them first for my yocto > >>>>>>>>>> environment. If you need something special please tell me. > >>>>>>>>>> > >>>>>>>>>>> This is Apache 2.4.x? > >>>>>>>>>> > >>>>>>>>>> yes it is Apache 2.4.16 > >>>>>>>>>> > >>>>>>>>>>> Is it failing on a request or on startup? > >>>>>>>>>> > >>>>>>>>>> its failing on every https request. > >>>>>>>>> > >>>>>>>>> strace in this case isn't particularly helpful as it doesn't show where > >>>>>>>>> it is crashing. > >>>>>>>>> > >>>>>>>>> Can I see your nss.conf? > >>>>>>>>> > >>>>>>>>> What version of NSS are you using? > >>>>>>>> > >>>>>>>> I'am using nss in version 3.19.2 > >>>>>>>> > >>>>>>>> here my nss.conf > >>>>>>>> > >>>>>>>> # > >>>>>>>> # This is the Apache server configuration file providing SSL support using. > >>>>>>>> # the mod_nss plugin. It contains the configuration directives to instruct > >>>>>>>> # the server how to serve pages over an https connection. > >>>>>>>> # > >>>>>>>> # Do NOT simply read the instructions in here without understanding > >>>>>>>> # what they do. They're here only as hints or reminders. If you are unsure > >>>>>>>> # consult the online docs. You have been warned. > >>>>>>>> # > >>>>>>>> > >>>>>>>> # > >>>>>>>> # When we also provide SSL we have to listen to the > >>>>>>>> # standard HTTP port (see above) and to the HTTPS port > >>>>>>>> # > >>>>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two > >>>>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" > >>>>>>>> # > >>>>>>>> Listen 443 > >>>>>>>> > >>>>>>>> ## > >>>>>>>> ## SSL Global Context > >>>>>>>> ## > >>>>>>>> ## All SSL configuration in this context applies both to > >>>>>>>> ## the main server and all SSL-enabled virtual hosts. > >>>>>>>> ## > >>>>>>>> > >>>>>>>> # > >>>>>>>> # Some MIME-types for downloading Certificates and CRLs > >>>>>>>> # > >>>>>>>> AddType application/x-x509-ca-cert .crt > >>>>>>>> AddType application/x-pkcs7-crl .crl > >>>>>>>> > >>>>>>>> > >>>>>>>> # Pass Phrase Helper: > >>>>>>>> # This helper program stores the token password pins between > >>>>>>>> # restarts of Apache. > >>>>>>>> # Unfortunately the directive is required even if we use no password > >>>>>>>> # (though in such case the program should never be used) > >>>>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache > >>>>>>>> > >>>>>>>> # Pass Phrase Dialog: > >>>>>>>> # Configure the pass phrase gathering process. > >>>>>>>> # The filtering dialog program (`builtin' is a internal > >>>>>>>> # terminal dialog) has to provide the pass phrase on stdout. > >>>>>>>> #NSSPassPhraseDialog builtin > >>>>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf > >>>>>>>> > >>>>>>>> # Configure the SSL Session Cache. > >>>>>>>> # NSSSessionCacheSize is the number of entries in the cache. > >>>>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). > >>>>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). > >>>>>>>> NSSSessionCacheSize 10000 > >>>>>>>> NSSSessionCacheTimeout 100 > >>>>>>>> NSSSession3CacheTimeout 86400 > >>>>>>>> > >>>>>>>> #RFC 5077 > >>>>>>>> NSSSessionTickets off > >>>>>>> > >>>>>>> SessionTickets are on not off > >>>>>>> > >>>>>>> NSSSessionTickets on > >>>>>>> > >>>>>> > >>>>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss > >>>>>> parameters for every virtual host. It seems that the problem is here. Because > >>>>>> if I remove the include of httpd-vhosts.conf no segfaults occur. > >>>>>> > >>>>>> here my httpd-vhosts.conf with nss related parameters. Are these > >>>>>> parameters capable for a virtual host configuration? > >>>>> > >>>>> You've included some directives that are expected to be global: > >>>>> > >>>>> NSSPassPhraseHelper > >>>>> NSSPassPhraseDialog > >>>>> NSSSession*Cache* > >>>>> NSSCertificateDatabase > >>>>> > >>>>> I doubt they hurt anything. > >>>> > >>>> ok thx, I removed these directives from the httpd-vhosts.conf > >>>> > >>>>> > >>>>> I still can't duplicate a crash. I ran it through valgrind to see if > >>>>> there was a memory issue and came up with nothing too. > >>>>> > >>>>> This could be an architecture problem that I just can't see on x86 > >>>>> hardware I suppose. > >>>> > >>>> I'am using a arm hardware (imx6) > >>>> > >>>>> This particular crash seems rather strange too. Within mod_nss enabling > >>>>> this just calls: > >>>>> > >>>>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, > >>>>> mctx->sc->session_tickets); > >>>>> > >>>>> Basically enabling it in the model server socket. I would imagine the > >>>>> crash is probably deeper in NSS. > >>>> > >>>> > >>>>> To narrow things down you might try the NSS tool selfserv. It has an > >>>>> option, -u, to enable session tickets. It might eliminate mod_nss as the > >>>>> crash source (or it might implicate it too). > >>>> > >>>> > >>>> thx for the hint to the NSS selfserv tool. > >>>> > >>>> I'am using it with DSA Keys instead of Eliptic Curve Keys because > >>>> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) > >>>> > >>>> First I generate a new nss db. > >>>> > >>>> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >>>> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >>>> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >>>> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >>>> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf > >>>> > >>>> Then starting selfserv with params: > >>>> > >>>> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u > >>>> > >>>> selfserv: About to call accept. > >>>> > >>>> Now I perform a https request with the Chrome Browser > >>>> > >>>> GET /test.php HTTP/1.1 > >>>> Host: 192.168.1.229 > >>>> Connection: keep-alive > >>>> Cache-Control: max-age=0 > >>>> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 > >>>> Upgrade-Insecure-Requests: 1 > >>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 > >>>> DNT: 1 > >>>> Accept-Encoding: gzip, deflate, sdch > >>>> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 > >>>> > >>>> EOF > >>>> > >>>> it looks for me that it works with selfserv. But I'm not sure if the Session > >>>> Ticket works and if it makes a difference not using ECC Keys. > >>> > >>> I checked with wireshark, and I saw the TLS Session Ticket Extension > >>> in the hello packet there. > >>> > >>> I'also switched back to my old ciphers in my apache setup nss.conf > >>> > >>> NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha > >>> > >>> and this is working together with NSSSessionTickets on. > >>> > >>> So it breaks if i'am using ECC Ciphersuite together with Session Tickets! > >>> > >>> NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256 > >>> NSSSessionTickets on > >>> > >>> is this something that can be explained? > >> > >> Hmm, interesting. It gives me something to look at. I was in the middle > >> of responding to your previous message when this one came in. > >> > >> I had been testing with an RSA key. Let me try an ECC key to see if I > >> can reproduce the crash (no success yet). > >> > >> Here is the previous bit I was going to send: > >> > >> This actually generated an RSA key for the cert you used for selfserv. > >> certutil will generate a certificate if the -k option isn't provided. > >> > >> You can do the same with ECC by adding to the last command -k ec -q > > > > > > I updated from nss 3.19 to nss 3.21 but I run into the same issue with it. > > > > Then I played around with gdb and saw that it breaks in libssl3.so > > Some hints how I can debug this further? > > > > Best Regards, > > > > Oliver > > > > gdb httpd > > run -X -e debug -D /usr/sbin/httpd > > > > root at ibis-box:/var/apache2/logs# gdb httpd > > GNU gdb (GDB) 7.9.1 > > Copyright (C) 2015 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > > and "show warranty" for details. > > This GDB was configured as "arm-poky-linux-gnueabi". > > Type "show configuration" for configuration details. > > For bug reporting instructions, please see: > > . > > Find the GDB manual and other documentation resources online at: > > . > > For help, type "help". > > Type "apropos word" to search for commands related to "word"... > > Reading symbols from httpd...(no debugging symbols found)...done. > > (gdb) run -X -e debug -D /usr/sbin/httpd > > Starting program: /usr/sbin/httpd -X -e debug -D /usr/sbin/httpd > > warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". > > To enable execution of this file add > > add-auto-load-safe-path /lib/libthread_db-1.0.so > > line to your configuration file "/home/root/.gdbinit". > > To completely disable this security protection add > > set auto-load safe-path / > > line to your configuration file "/home/root/.gdbinit". > > For more information about this security protection see the > > "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: > > info "(gdb)Auto-loading safe path" > > warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. > > [Fri Feb 26 00:03:18.648483 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so > > [Fri Feb 26 00:03:18.700565 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so > > [Fri Feb 26 00:03:18.750258 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so > > [Fri Feb 26 00:03:18.801825 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so > > [Fri Feb 26 00:03:18.852520 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so > > [Fri Feb 26 00:03:18.909582 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so > > [Fri Feb 26 00:03:18.964388 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so > > [Fri Feb 26 00:03:19.022456 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so > > [Fri Feb 26 00:03:19.081787 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so > > [Fri Feb 26 00:03:19.142737 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so > > [Fri Feb 26 00:03:19.205120 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so > > [Fri Feb 26 00:03:19.288557 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so > > [Fri Feb 26 00:03:19.355114 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so > > [Fri Feb 26 00:03:19.428930 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module log_config_module from /usr/lib/apache2/modules/mod_log_config.so > > [Fri Feb 26 00:03:19.495578 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so > > [Fri Feb 26 00:03:19.569779 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so > > [Fri Feb 26 00:03:19.641764 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so > > [Fri Feb 26 00:03:19.716852 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module version_module from /usr/lib/apache2/modules/mod_version.so > > [Fri Feb 26 00:03:20.213946 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module nss_module from /usr/lib/apache2/modules/libmodnss.so > > [Fri Feb 26 00:03:20.307607 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so > > [Fri Feb 26 00:03:20.393425 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module unixd_module from /usr/lib/apache2/modules/mod_unixd.so > > [Fri Feb 26 00:03:20.483090 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so > > [Fri Feb 26 00:03:20.578632 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so > > [Fri Feb 26 00:03:20.671102 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so > > [Fri Feb 26 00:03:20.763080 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so > > > > Program received signal SIGSEGV, Segmentation fault. > > 0x76b366dc in ?? () from /usr/lib/libssl3.so > > Can I ask why you have NSSEnforceValidCerts off set? Is it because of an > error that was logged, preventing the server from starting? yes because of this error: Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved. if I also set NSSECCNickname "xxxxxxx" I got: SSL Library Error: -8172 Certificate is signed by an untrusted issuer Best Regards, Oliver From oliver.graute at gmail.com Mon Feb 29 08:27:58 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Mon, 29 Feb 2016 09:27:58 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56D07029.7050202@redhat.com> References: <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> <20160226131912.GD29619@graute-opti> <56D07029.7050202@redhat.com> Message-ID: <20160229082758.GB14634@graute-opti> On 26/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > On 25/02/16, Rob Crittenden wrote: > >> Oliver Graute wrote: > >>> On 25/02/16, Oliver Graute wrote: > >>>> On 24/02/16, Rob Crittenden wrote: > >>>>> Oliver Graute wrote: > >>>>>> On 23/02/16, Oliver Graute wrote: > >>>>>>> On Tue, Feb 23, 2016 at 5:10 PM, Oliver Graute wrote: > >>>>>>>> On 23/02/16, Rob Crittenden wrote: > >>>>>>>>> Oliver Graute wrote: > >>>>>>>>>> On 22/02/16, Rob Crittenden wrote: > >>>>>>>>>>> Oliver Graute wrote: > >>>>>>>>>>>> Hello, > >>>>>>>>>>>> > >>>>>>>>>>>> I installed the mod_nss plugin in version 1.0.12 on my apache webserver, > >>>>>>>>>>>> TLS on Port 443 is working fine until I enable the new NSSSession ticket > >>>>>>>>>>>> feature in my nss.conf with: > >>>>>>>>>>>> > >>>>>>>>>>>> #RFC 5077 > >>>>>>>>>>>> NSSSessionTickets on > >>>>>>>>>>>> > >>>>>>>>>>>> then something is broken, I see segfaults in my apache error log: > >>>>>>>>>>>> > >>>>>>>>>>>> [Fri Feb 19 10:12:15.338660 2016] [mpm_prefork:notice] [pid 413] AH00163: Apache/2.4.16 (Unix) mod_nss/1.0.12 NSS/3.19.2 Basic ECC PHP/5.5.10 configured -- resuming normal operations > >>>>>>>>>>>> [Fri Feb 19 10:12:15.338843 2016] [mpm_prefork:info] [pid 413] AH00164: Server built: Feb 22 2016 12:44:38 > >>>>>>>>>>>> [Fri Feb 19 10:12:15.339046 2016] [core:notice] [pid 413] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND -D SSL -D PHP5' > >>>>>>>>>>>> [Fri Feb 19 10:12:15.339160 2016] [mpm_prefork:debug] [pid 413] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > >>>>>>>>>>>> [Fri Feb 19 10:12:15.386483 2016] [:debug] [pid 416] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>>>> [Fri Feb 19 10:12:15.386853 2016] [:info] [pid 416] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>>>> [Fri Feb 19 10:12:40.374175 2016] [core:notice] [pid 413] AH00052: child pid 416 exit signal Segmentation fault (11) > >>>>>>>>>>>> [Fri Feb 19 10:12:41.496820 2016] [:debug] [pid 423] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>>>> [Fri Feb 19 10:12:41.497224 2016] [:info] [pid 423] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>>>> [Fri Feb 19 10:12:42.388948 2016] [core:notice] [pid 413] AH00052: child pid 423 exit signal Segmentation fault (11) > >>>>>>>>>>>> [Fri Feb 19 10:12:43.508779 2016] [:debug] [pid 424] nss_engine_init.c(286): SNI is enabled > >>>>>>>>>>>> [Fri Feb 19 10:12:43.509217 2016] [:info] [pid 424] Init: Seeding PRNG with 136 bytes of entropy > >>>>>>>>>>>> [Fri Feb 19 10:12:44.404130 2016] [core:notice] [pid 413] AH00052: child pid 424 exit signal Segmentation fault (11) > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> and in Chrome Browser I got: > >>>>>>>>>>>> > >>>>>>>>>>>> ERR_SSL_VERSION_OR_CIPHER_MISMATCH > >>>>>>>>>>>> > >>>>>>>>>>>> I tested also a basic ssl client connection with openssl: > >>>>>>>>>>>> > >>>>>>>>>>>> openssl s_client -connect 192.168.1.229:443 -state -debug > >>>>>>>>>>>> > >>>>>>>>>>>> SSL_connect:SSLv3 read server certificate A > >>>>>>>>>>>> SSL_connect:SSLv3 read server key exchange A > >>>>>>>>>>>> SSL_connect:SSLv3 read server done A > >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (75 bytes => 75 (0x4B)) > >>>>>>>>>>>> 0000 - 16 03 03 00 46 10 00 00-42 41 04 3d c7 93 63 45 ....F...BA.=..cE > >>>>>>>>>>>> 0010 - 79 41 11 bc 06 c0 b7 c6-d1 b5 33 d9 86 a6 d5 e9 yA........3..... > >>>>>>>>>>>> 0020 - 36 e4 2b ac 0e bc 70 d6-d6 8c a7 a9 3c dd 1b 0c 6.+...p.....<... > >>>>>>>>>>>> 0030 - 77 48 20 38 dd 1e c9 a1-05 6c 5c b6 c9 f4 99 f2 wH 8.....l\..... > >>>>>>>>>>>> 0040 - 1a 18 ae 81 63 71 65 90-e8 a5 b6 ....cqe.... > >>>>>>>>>>>> SSL_connect:SSLv3 write client key exchange A > >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (6 bytes => 6 (0x6)) > >>>>>>>>>>>> 0000 - 14 03 03 00 01 01 ...... > >>>>>>>>>>>> SSL_connect:SSLv3 write change cipher spec A > >>>>>>>>>>>> write to 0x205dec0 [0x206dd50] (45 bytes => 45 (0x2D)) > >>>>>>>>>>>> 0000 - 16 03 03 00 28 b1 e0 60-8a 2c 97 cf a0 4f 97 ee ....(..`.,...O.. > >>>>>>>>>>>> 0010 - cd 8f 05 41 aa 50 a6 73-a3 4c 86 1e 5f 3c 7b 2b ...A.P.s.L.._<{+ > >>>>>>>>>>>> 0020 - 2d 7e 6a 68 dc 97 94 9d-91 15 c0 0e 27 -~jh........' > >>>>>>>>>>>> SSL_connect:SSLv3 write finished A > >>>>>>>>>>>> SSL_connect:SSLv3 flush data > >>>>>>>>>>>> read from 0x205dec0 [0x2063f83] (5 bytes => 0 (0x0)) > >>>>>>>>>>>> SSL_connect:failed in SSLv3 read server session ticket A > >>>>>>>>>>>> 140123095688864:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: > >>>>>>>>>>>> > >>>>>>>>>>>> apache and mod_nss are build from the sources for an embedded yocto environment. > >>>>>>>>>>>> > >>>>>>>>>>>> some ideas, whats going on here? > >>>>>>>>>>> > >>>>>>>>>>> Can you get a stack trace from the core? > >>>>>>>>>> > >>>>>>>>>> I can give you an strace, see below. Other stack tools are currently not > >>>>>>>>>> available, because I need to compile them first for my yocto > >>>>>>>>>> environment. If you need something special please tell me. > >>>>>>>>>> > >>>>>>>>>>> This is Apache 2.4.x? > >>>>>>>>>> > >>>>>>>>>> yes it is Apache 2.4.16 > >>>>>>>>>> > >>>>>>>>>>> Is it failing on a request or on startup? > >>>>>>>>>> > >>>>>>>>>> its failing on every https request. > >>>>>>>>> > >>>>>>>>> strace in this case isn't particularly helpful as it doesn't show where > >>>>>>>>> it is crashing. > >>>>>>>>> > >>>>>>>>> Can I see your nss.conf? > >>>>>>>>> > >>>>>>>>> What version of NSS are you using? > >>>>>>>> > >>>>>>>> I'am using nss in version 3.19.2 > >>>>>>>> > >>>>>>>> here my nss.conf > >>>>>>>> > >>>>>>>> # > >>>>>>>> # This is the Apache server configuration file providing SSL support using. > >>>>>>>> # the mod_nss plugin. It contains the configuration directives to instruct > >>>>>>>> # the server how to serve pages over an https connection. > >>>>>>>> # > >>>>>>>> # Do NOT simply read the instructions in here without understanding > >>>>>>>> # what they do. They're here only as hints or reminders. If you are unsure > >>>>>>>> # consult the online docs. You have been warned. > >>>>>>>> # > >>>>>>>> > >>>>>>>> # > >>>>>>>> # When we also provide SSL we have to listen to the > >>>>>>>> # standard HTTP port (see above) and to the HTTPS port > >>>>>>>> # > >>>>>>>> # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two > >>>>>>>> # Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443" > >>>>>>>> # > >>>>>>>> Listen 443 > >>>>>>>> > >>>>>>>> ## > >>>>>>>> ## SSL Global Context > >>>>>>>> ## > >>>>>>>> ## All SSL configuration in this context applies both to > >>>>>>>> ## the main server and all SSL-enabled virtual hosts. > >>>>>>>> ## > >>>>>>>> > >>>>>>>> # > >>>>>>>> # Some MIME-types for downloading Certificates and CRLs > >>>>>>>> # > >>>>>>>> AddType application/x-x509-ca-cert .crt > >>>>>>>> AddType application/x-pkcs7-crl .crl > >>>>>>>> > >>>>>>>> > >>>>>>>> # Pass Phrase Helper: > >>>>>>>> # This helper program stores the token password pins between > >>>>>>>> # restarts of Apache. > >>>>>>>> # Unfortunately the directive is required even if we use no password > >>>>>>>> # (though in such case the program should never be used) > >>>>>>>> NSSPassPhraseHelper /usr/lib/apache2/bin/nss_pcache > >>>>>>>> > >>>>>>>> # Pass Phrase Dialog: > >>>>>>>> # Configure the pass phrase gathering process. > >>>>>>>> # The filtering dialog program (`builtin' is a internal > >>>>>>>> # terminal dialog) has to provide the pass phrase on stdout. > >>>>>>>> #NSSPassPhraseDialog builtin > >>>>>>>> NSSPassPhraseDialog file:/etc/apache2/password.conf > >>>>>>>> > >>>>>>>> # Configure the SSL Session Cache. > >>>>>>>> # NSSSessionCacheSize is the number of entries in the cache. > >>>>>>>> # NSSSessionCacheTimeout is the SSL2 session timeout (in seconds). > >>>>>>>> # NSSSession3CacheTimeout is the SSL3/TLS session timeout (in seconds). > >>>>>>>> NSSSessionCacheSize 10000 > >>>>>>>> NSSSessionCacheTimeout 100 > >>>>>>>> NSSSession3CacheTimeout 86400 > >>>>>>>> > >>>>>>>> #RFC 5077 > >>>>>>>> NSSSessionTickets off > >>>>>>> > >>>>>>> SessionTickets are on not off > >>>>>>> > >>>>>>> NSSSessionTickets on > >>>>>>> > >>>>>> > >>>>>> in my /etc/apache2/extra/httpd-vhosts.conf I also added some mod_nss > >>>>>> parameters for every virtual host. It seems that the problem is here. Because > >>>>>> if I remove the include of httpd-vhosts.conf no segfaults occur. > >>>>>> > >>>>>> here my httpd-vhosts.conf with nss related parameters. Are these > >>>>>> parameters capable for a virtual host configuration? > >>>>> > >>>>> You've included some directives that are expected to be global: > >>>>> > >>>>> NSSPassPhraseHelper > >>>>> NSSPassPhraseDialog > >>>>> NSSSession*Cache* > >>>>> NSSCertificateDatabase > >>>>> > >>>>> I doubt they hurt anything. > >>>> > >>>> ok thx, I removed these directives from the httpd-vhosts.conf > >>>> > >>>>> > >>>>> I still can't duplicate a crash. I ran it through valgrind to see if > >>>>> there was a memory issue and came up with nothing too. > >>>>> > >>>>> This could be an architecture problem that I just can't see on x86 > >>>>> hardware I suppose. > >>>> > >>>> I'am using a arm hardware (imx6) > >>>> > >>>>> This particular crash seems rather strange too. Within mod_nss enabling > >>>>> this just calls: > >>>>> > >>>>> SSL_OptionSet(mctx->model, SSL_ENABLE_SESSION_TICKETS, > >>>>> mctx->sc->session_tickets); > >>>>> > >>>>> Basically enabling it in the model server socket. I would imagine the > >>>>> crash is probably deeper in NSS. > >>>> > >>>> > >>>>> To narrow things down you might try the NSS tool selfserv. It has an > >>>>> option, -u, to enable session tickets. It might eliminate mod_nss as the > >>>>> crash source (or it might implicate it too). > >>>> > >>>> > >>>> thx for the hint to the NSS selfserv tool. > >>>> > >>>> I'am using it with DSA Keys instead of Eliptic Curve Keys because > >>>> selfserf can't handle ECC. (selfserv -Y didn't show me ECC ciphers) > >>>> > >>>> First I generate a new nss db. > >>>> > >>>> certutil -N -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >>>> certutil -G -k dsa -n localhost -t "TC,," -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >>>> certutil -K -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >>>> certutil -L -d /etc/apache2/nss-conf -f /etc/apache2/password2.conf > >>>> certutil -S -s "CN=localhost" -x -n localhost -t "TC,," -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf > >>>> > >>>> Then starting selfserv with params: > >>>> > >>>> selfserv -n "localhost" -V tls1.2: -d /etc/apache2/nss-conf/ -f /etc/apache2/password2.conf -p 443 -v -u > >>>> > >>>> selfserv: About to call accept. > >>>> > >>>> Now I perform a https request with the Chrome Browser > >>>> > >>>> GET /test.php HTTP/1.1 > >>>> Host: 192.168.1.229 > >>>> Connection: keep-alive > >>>> Cache-Control: max-age=0 > >>>> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 > >>>> Upgrade-Insecure-Requests: 1 > >>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 > >>>> DNT: 1 > >>>> Accept-Encoding: gzip, deflate, sdch > >>>> Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 > >>>> > >>>> EOF > >>>> > >>>> it looks for me that it works with selfserv. But I'm not sure if the Session > >>>> Ticket works and if it makes a difference not using ECC Keys. > >>> > >>> I checked with wireshark, and I saw the TLS Session Ticket Extension > >>> in the hello packet there. > >>> > >>> I'also switched back to my old ciphers in my apache setup nss.conf > >>> > >>> NSSCipherSuite +rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha > >>> > >>> and this is working together with NSSSessionTickets on. > >>> > >>> So it breaks if i'am using ECC Ciphersuite together with Session Tickets! > >>> > >>> NSSCipherSuite +ecdhe_ecdsa_aes_128_gcm_sha256,+ecdhe_ecdsa_aes_128_sha256 > >>> NSSSessionTickets on > >>> > >>> is this something that can be explained? > >> > >> Hmm, interesting. It gives me something to look at. I was in the middle > >> of responding to your previous message when this one came in. > >> > >> I had been testing with an RSA key. Let me try an ECC key to see if I > >> can reproduce the crash (no success yet). > >> > >> Here is the previous bit I was going to send: > >> > >> This actually generated an RSA key for the cert you used for selfserv. > >> certutil will generate a certificate if the -k option isn't provided. > >> > >> You can do the same with ECC by adding to the last command -k ec -q > > > > > > I updated from nss 3.19 to nss 3.21 but I run into the same issue with it. > > > > Then I played around with gdb and saw that it breaks in libssl3.so > > Some hints how I can debug this further? > > > > Best Regards, > > > > Oliver > > > > gdb httpd > > run -X -e debug -D /usr/sbin/httpd > > > > root at ibis-box:/var/apache2/logs# gdb httpd > > GNU gdb (GDB) 7.9.1 > > Copyright (C) 2015 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > > and "show warranty" for details. > > This GDB was configured as "arm-poky-linux-gnueabi". > > Type "show configuration" for configuration details. > > For bug reporting instructions, please see: > > . > > Find the GDB manual and other documentation resources online at: > > . > > For help, type "help". > > Type "apropos word" to search for commands related to "word"... > > Reading symbols from httpd...(no debugging symbols found)...done. > > (gdb) run -X -e debug -D /usr/sbin/httpd > > Starting program: /usr/sbin/httpd -X -e debug -D /usr/sbin/httpd > > warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". > > To enable execution of this file add > > add-auto-load-safe-path /lib/libthread_db-1.0.so > > line to your configuration file "/home/root/.gdbinit". > > To completely disable this security protection add > > set auto-load safe-path / > > line to your configuration file "/home/root/.gdbinit". > > For more information about this security protection see the > > "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: > > info "(gdb)Auto-loading safe path" > > warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. > > [Fri Feb 26 00:03:18.648483 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so > > [Fri Feb 26 00:03:18.700565 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so > > [Fri Feb 26 00:03:18.750258 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so > > [Fri Feb 26 00:03:18.801825 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_groupfile_module from /usr/lib/apache2/modules/mod_authz_groupfile.so > > [Fri Feb 26 00:03:18.852520 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so > > [Fri Feb 26 00:03:18.909582 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so > > [Fri Feb 26 00:03:18.964388 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so > > [Fri Feb 26 00:03:19.022456 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so > > [Fri Feb 26 00:03:19.081787 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module socache_shmcb_module from /usr/lib/apache2/modules/mod_socache_shmcb.so > > [Fri Feb 26 00:03:19.142737 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so > > [Fri Feb 26 00:03:19.205120 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so > > [Fri Feb 26 00:03:19.288557 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so > > [Fri Feb 26 00:03:19.355114 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so > > [Fri Feb 26 00:03:19.428930 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module log_config_module from /usr/lib/apache2/modules/mod_log_config.so > > [Fri Feb 26 00:03:19.495578 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so > > [Fri Feb 26 00:03:19.569779 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module headers_module from /usr/lib/apache2/modules/mod_headers.so > > [Fri Feb 26 00:03:19.641764 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so > > [Fri Feb 26 00:03:19.716852 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module version_module from /usr/lib/apache2/modules/mod_version.so > > [Fri Feb 26 00:03:20.213946 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module nss_module from /usr/lib/apache2/modules/libmodnss.so > > [Fri Feb 26 00:03:20.307607 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module mpm_prefork_module from /usr/lib/apache2/modules/mod_mpm_prefork.so > > [Fri Feb 26 00:03:20.393425 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module unixd_module from /usr/lib/apache2/modules/mod_unixd.so > > [Fri Feb 26 00:03:20.483090 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so > > [Fri Feb 26 00:03:20.578632 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so > > [Fri Feb 26 00:03:20.671102 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so > > [Fri Feb 26 00:03:20.763080 2016] [so:debug] [pid 441] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so > > > > Program received signal SIGSEGV, Segmentation fault. > > 0x76b366dc in ?? () from /usr/lib/libssl3.so > > If you could build NSS with debugging we would be able to see where it > is failing. I'm not sure where your package is coming from but IIRC to > build with debug you can pass in something at build time, like BUILD_OPT=0 i'am using this yocto recipe: https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-support/nss/nss_3.21.bb I changed the option BUILD_OPT=0 and rebuild the lib, how can I see the debug stuff? Best regards, Oliver From rcritten at redhat.com Mon Feb 29 14:53:37 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 29 Feb 2016 09:53:37 -0500 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160229082758.GB14634@graute-opti> References: <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> <20160226131912.GD29619@graute-opti> <56D07029.7050202@redhat.com> <20160229082758.GB14634@graute-opti> Message-ID: <56D45B71.2020704@redhat.com> Oliver Graute wrote: > On 26/02/16, Rob Crittenden wrote: >> >> If you could build NSS with debugging we would be able to see where it >> is failing. I'm not sure where your package is coming from but IIRC to >> build with debug you can pass in something at build time, like BUILD_OPT=0 > > i'am using this yocto recipe: > > https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-support/nss/nss_3.21.bb > > I changed the option > > BUILD_OPT=0 > > and rebuild the lib, how can I see the debug stuff? I'd suggest generating new core then open that in gdb. With the updated NSS you should have more details on where it crashed. rob From rcritten at redhat.com Mon Feb 29 14:55:49 2016 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 29 Feb 2016 09:55:49 -0500 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <20160229081817.GA14634@graute-opti> References: <56CC6202.9030700@redhat.com> <20160223161009.GA8558@graute-opti> <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> <20160226131912.GD29619@graute-opti> <56D0A72C.2010506@redhat.com> <20160229081817.GA14634@graute-opti> Message-ID: <56D45BF5.70802@redhat.com> Oliver Graute wrote: > On 26/02/16, Rob Crittenden wrote: >> Can I ask why you have NSSEnforceValidCerts off set? Is it because of an >> error that was logged, preventing the server from starting? > > yes because of this error: > > Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved. > > if I also set NSSECCNickname "xxxxxxx" I got: > > SSL Library Error: -8172 Certificate is signed by an untrusted issuer Ok, that shouldn't cause any crash problems. I was wondering if you were getting bad key usage flags or anything else. This should be fixable by adding the issuing CA to the Apache NSS database using certutil. rob From oliver.graute at gmail.com Mon Feb 29 15:15:05 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Mon, 29 Feb 2016 16:15:05 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56D45BF5.70802@redhat.com> References: <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> <20160226131912.GD29619@graute-opti> <56D0A72C.2010506@redhat.com> <20160229081817.GA14634@graute-opti> <56D45BF5.70802@redhat.com> Message-ID: <20160229151505.GA16783@graute-opti> On 29/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > On 26/02/16, Rob Crittenden wrote: > >> Can I ask why you have NSSEnforceValidCerts off set? Is it because of an > >> error that was logged, preventing the server from starting? > > > > yes because of this error: > > > > Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved. > > > > if I also set NSSECCNickname "xxxxxxx" I got: > > > > SSL Library Error: -8172 Certificate is signed by an untrusted issuer > > Ok, that shouldn't cause any crash problems. I was wondering if you were > getting bad key usage flags or anything else. > > This should be fixable by adding the issuing CA to the Apache NSS > database using certutil. yes you are right, I fixed it by adding the CA to the database and setting the Trusted Attributes to the right values with: certutils -M -n "CA Nickname" -t "CT,C,c" Best regards, Oliver From oliver.graute at gmail.com Mon Feb 29 15:42:55 2016 From: oliver.graute at gmail.com (Oliver Graute) Date: Mon, 29 Feb 2016 16:42:55 +0100 Subject: [Mod_nss-list] NSSSessionTickets causes some segfault in error log In-Reply-To: <56D45B71.2020704@redhat.com> References: <20160224100216.GA29619@graute-opti> <56CDD887.1000101@redhat.com> <20160225141039.GB29619@graute-opti> <20160225151241.GC29619@graute-opti> <56CF1ECC.4080303@redhat.com> <20160226131912.GD29619@graute-opti> <56D07029.7050202@redhat.com> <20160229082758.GB14634@graute-opti> <56D45B71.2020704@redhat.com> Message-ID: <20160229154255.GB16783@graute-opti> On 29/02/16, Rob Crittenden wrote: > Oliver Graute wrote: > > On 26/02/16, Rob Crittenden wrote: > >> > >> If you could build NSS with debugging we would be able to see where it > >> is failing. I'm not sure where your package is coming from but IIRC to > >> build with debug you can pass in something at build time, like BUILD_OPT=0 > > > > i'am using this yocto recipe: > > > > https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-support/nss/nss_3.21.bb > > > > I changed the option > > > > BUILD_OPT=0 > > > > and rebuild the lib, how can I see the debug stuff? > > I'd suggest generating new core then open that in gdb. With the updated > NSS you should have more details on where it crashed. ok I generated a new core and debugged it with gdb. But i'am a bit stucked here because my gdb can't do thread debugging Unable to find libthread_db matching inferior's thread library, thread debugging will not be available I already added add-auto-load-safe-path /lib/libthread_db-1.0.so to my .gdbinit to fix that, but no success. all my gdb says is: Program received signal SIGSEGV, Segmentation fault. 0x76b366dc in ?? () from /usr/lib/libssl3.so Best Regards, Oliver