From nagemnna at gmail.com Wed Jun 3 14:31:56 2015 From: nagemnna at gmail.com (Megan .) Date: Wed, 3 Jun 2015 10:31:56 -0400 Subject: [Linux-cluster] Error: ClientSocket(String): connect() failed: No such file or directory Message-ID: Anybody ever seen "Error: ClientSocket(String): connect() failed: No such file or directory" when doing a start all? Something seems to have broken with our closer. Our UAT setup works as expected. I looked at tcpdumps the best that i could (i'm not a network person though) and i didn't see anything obvious. I shutdown iptables on all nodes. We are running Centos 6,6, ccs-0.16.2-75.el6_6.1.x86_64 cman-3.0.12.1-68.el6.x86_64. We have a 12 node cluster in production that allows us to share gfs2 iscsi mounts. no other services are used. clvmd -R runs fine at this time. ccs -h node --sync --activate also runs fine. [root at admin1 ~]# ccs -h admin1-ops --startall Unable to start map1-ops, possibly due to lack of quorum, try --startall Error: ClientSocket(String): connect() failed: No such file or directory Started cache2-ops Unable to start data1-ops, possibly due to lack of quorum, try --startall Error: ClientSocket(String): connect() failed: No such file or directory Started map2-ops Unable to start archive1-ops, possibly due to lack of quorum, try --startall Error: ClientSocket(String): connect() failed: No such file or directory Started data3-ops Started mgmt1-ops Unable to start admin1-ops, possibly due to lack of quorum, try --startall Error: ClientSocket(String): connect() failed: No such file or directory Started data2-ops Started cache1-ops [root at admin1 ~]# I have quorum: [root at admin1 ~]# clustat Cluster Status for bitsops @ Wed Jun 3 02:13:08 2015 Member Status: Quorate Member Name ID Status ------ ---- ---- ------ admin1-ops 1 Online, Local mgmt1-ops 2 Online archive1-ops 3 Online map1-ops 4 Online map2-ops 5 Online cache1-ops 6 Online cache2-ops 7 Online data1-ops 8 Online data2-ops 9 Online data3-ops 10 Online Here is what I expect, and what UAT gives me: [root at admin1-uat ~]# ccs -h admin1-uat --startall Started mgmt1-uat Started data1-uat Started data2-uat Started admin1-uat Started tools-uat Started map1-uat Started archive1-uat Started cache2-uat Started cache1-uat Started map2-uat [root at admin1-uat ~]# -------------- next part -------------- An HTML attachment was scrubbed... URL: From nagemnna at gmail.com Thu Jun 4 12:23:55 2015 From: nagemnna at gmail.com (Megan .) Date: Thu, 4 Jun 2015 08:23:55 -0400 Subject: [Linux-cluster] Error: ClientSocket(String): connect() failed: No such file or directory In-Reply-To: References: Message-ID: FYI - i talked to our network folks and it looks like they were doing some testing last night with port failover which may or may not have caused this issue. However, I was able to correct it by fencing the problem nodes. On Wed, Jun 3, 2015 at 10:31 AM, Megan . wrote: > Anybody ever seen "Error: ClientSocket(String): connect() failed: No such > file or directory" when doing a start all? Something seems to have > broken with our closer. Our UAT setup works as expected. I looked at > tcpdumps the best that i could (i'm not a network person though) and i > didn't see anything obvious. I shutdown iptables on all nodes. > > We are running Centos 6,6, ccs-0.16.2-75.el6_6.1.x86_64 > cman-3.0.12.1-68.el6.x86_64. We have a 12 node cluster in production that > allows us to share gfs2 iscsi mounts. no other services are used. clvmd > -R runs fine at this time. ccs -h node --sync --activate also runs fine. > > > [root at admin1 ~]# ccs -h admin1-ops --startall > > Unable to start map1-ops, possibly due to lack of quorum, try --startall > > Error: ClientSocket(String): connect() failed: No such file or directory > > Started cache2-ops > > Unable to start data1-ops, possibly due to lack of quorum, try --startall > > Error: ClientSocket(String): connect() failed: No such file or directory > > Started map2-ops > > Unable to start archive1-ops, possibly due to lack of quorum, try > --startall > > Error: ClientSocket(String): connect() failed: No such file or directory > > Started data3-ops > > Started mgmt1-ops > > Unable to start admin1-ops, possibly due to lack of quorum, try --startall > > Error: ClientSocket(String): connect() failed: No such file or directory > > Started data2-ops > > Started cache1-ops > > [root at admin1 ~]# > > I have quorum: > > [root at admin1 ~]# clustat > > Cluster Status for bitsops @ Wed Jun 3 02:13:08 2015 > > Member Status: Quorate > > > Member Name ID > Status > > ------ ---- ---- > ------ > > admin1-ops 1 > Online, Local > > mgmt1-ops 2 > Online > > archive1-ops 3 > Online > > map1-ops 4 > Online > > map2-ops 5 > Online > > cache1-ops 6 > Online > > cache2-ops 7 > Online > > data1-ops 8 > Online > > data2-ops 9 > Online > > data3-ops 10 > Online > > > > > Here is what I expect, and what UAT gives me: > > [root at admin1-uat ~]# ccs -h admin1-uat --startall > > Started mgmt1-uat > > Started data1-uat > > Started data2-uat > > Started admin1-uat > > Started tools-uat > > Started map1-uat > > Started archive1-uat > > Started cache2-uat > > Started cache1-uat > > Started map2-uat > > [root at admin1-uat ~]# > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpokorny at redhat.com Mon Jun 22 17:26:14 2015 From: jpokorny at redhat.com (Jan =?utf-8?Q?Pokorn=C3=BD?=) Date: Mon, 22 Jun 2015 19:26:14 +0200 Subject: [Linux-cluster] Error: ClientSocket(String): connect() failed: No such file or directory In-Reply-To: References: Message-ID: <20150622172614.GA25393@redhat.com> Hello Megan, On 04/06/15 08:23 -0400, Megan . wrote: > On Wed, Jun 3, 2015 at 10:31 AM, Megan . wrote: [...] > FYI - i talked to our network folks and it looks like they were doing some > testing last night with port failover which may or may not have caused this unlikely, unless you were "lucky" enough to contact a different actual machine under the network address than you intended or if modclusterd was fragile enough to break on these intermittent changes (not exactly sure what you mean with "port failover" TBH). Indicated error: >> Error: ClientSocket(String): connect() failed: No such file or directory means that modclusterd on particular node was not running (by itself, this is still OK) and it could not be started within 8 seconds, which is what modcluster (ricci's helper, but from clustermon package) tries to do if the socket /var/run/clumond.sock (indication of running modclusterd) cannot be reached (for whatever reason, including SELinux, but that should be OK as well). So if the problem recidivates, definitely check the troubling node if: - modclusterd service is running and/or is able to start (provide /var/run/clumond.sock socket) within 5 seconds or so under the typical workload (may be subtle in virtualized environment) - when modclusterd is started, /var/run/clumond.sock exists and has the expected properties (file-like socket, expected permissions) - SELinux (if enabled) audit contains any clumond.sock or modclusterd reference > issue. However, I was able to correct it by fencing the problem nodes. Provided that those "port failover" shakes were settled down by that time, perhaps modclusterd just started to be happy again and not failing anymore if it was the case previously. >> Anybody ever seen "Error: ClientSocket(String): connect() failed: No such >> file or directory" when doing a start all? Something seems to have >> broken with our closer. Our UAT setup works as expected. I looked at >> tcpdumps the best that i could (i'm not a network person though) and i >> didn't see anything obvious. I shutdown iptables on all nodes. FWIW, most if not all of the packet sniffing tools cannot hook into local file-like sockets. >> We are running Centos 6,6, ccs-0.16.2-75.el6_6.1.x86_64 Good, this excluded all known (and fixed!) bugs preventing modclusterd from operation (IPv4-only environment, huge cluster.conf). >> cman-3.0.12.1-68.el6.x86_64. We have a 12 node cluster in production that >> allows us to share gfs2 iscsi mounts. no other services are used. clvmd >> -R runs fine at this time. ccs -h node --sync --activate also runs fine. >> >> >> [root at admin1 ~]# ccs -h admin1-ops --startall >> Unable to start map1-ops, possibly due to lack of quorum, try --startall >> Error: ClientSocket(String): connect() failed: No such file or directory >> Started cache2-ops >> Unable to start data1-ops, possibly due to lack of quorum, try --startall >> Error: ClientSocket(String): connect() failed: No such file or directory >> Started map2-ops >> Unable to start archive1-ops, possibly due to lack of quorum, try >> --startall >> Error: ClientSocket(String): connect() failed: No such file or directory >> Started data3-ops >> Started mgmt1-ops >> Unable to start admin1-ops, possibly due to lack of quorum, try --startall >> Error: ClientSocket(String): connect() failed: No such file or directory >> Started data2-ops >> Started cache1-ops The out-of-context, hilarious hint (use --startall when you actually do) led me to file a bug: . Thanks for indirectly showing this off! -- Jan -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From daniel.dehennin at baby-gnu.org Tue Jun 30 19:37:27 2015 From: daniel.dehennin at baby-gnu.org (Daniel Dehennin) Date: Tue, 30 Jun 2015 21:37:27 +0200 Subject: [Linux-cluster] Finding the bottleneck between SAN and GFS2 Message-ID: <8761653s3s.fsf@hati.baby-gnu.org> Hello, We are experiencing slow VMs on our OpenNebula architecture: - two Dell PowerEdge M620 + Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz + 96GB RAM + 2x146Go SAS drives - 2TB SAN LUN to store qcow2 images with GFS2 over cLVM We made some tests, installing Linux OS in parallel and we did not find any issues with performance. Since 3 weeks, 17 users use ?60 VMs and everything became slow. The SAN administrator complain about very high IO/s so we limited each VM to 80 IO/s with the libvirt configuration #+begin_src xml 80 #+end_src But it did not get better Today I ran some benchmark to try to find out what happens. Checking plocks/s ================= I started with ping_pong[1] to see how many locks per second the GFS2 can sustain. I use it as describe on the samba wiki[2], here are the results: - starting ?ping_pong /var/lib/one/datastores/test_plock 3? on first node display around 4k plocks/s - then starting ?ping_pong /var/lib/one/datastores/test_plock 3? on the second node display around 2k on each node For the single node process, I was expecting an much higher rate, they speak about 500k to 1M locks/s. Do my numbers looks strange? Checking fileio =============== I use ?sysbench --test=fileio? to check inside the VM and outside (on bare metal node), with files in cache or cache dropped. The short result is that bare metal access to the GFS2 without any cache is terribly slow, around 2Mb/s and 90 requests/s. Is there a way to find out if the problem comes from my GFS2/corosync/pacemaker configuration or from the SAN? Regards. Following are the full sysbench results In the VM, qemu disk cache disabled, total_iops_sec = 0 ------------------------------------------------------- I try with the IO limit but the difference is minimal: - the request/s drop to ?80 - the Mb/s is around 1.2Mb/s root at vm:~# sysbench --num-threads=16 --test=fileio --file-total-size=9G --file-test-mode=rndrw prepare sysbench 0.4.12: multi-threaded system evaluation benchmark 128 files, 73728Kb each, 9216Mb total Creating files for the test... root at vm:~# sysbench --num-threads=16 --test=fileio --file-total-size=9G --file-test-mode=rndrw run sysbench 0.4.12: multi-threaded system evaluation benchmark Running the test with following options: Number of threads: 16 Extra file open flags: 0 128 files, 72Mb each 9Gb total file size Block size 16Kb Number of random requests for random IO: 10000 Read/Write ratio for combined random IO test: 1.50 Periodic FSYNC enabled, calling fsync() each 100 requests. Calling fsync() at the end of test, Enabled. Using synchronous I/O mode Doing random r/w test Threads started! Done. Operations performed: 6034 Read, 4019 Write, 12808 Other = 22861 Total Read 94.281Mb Written 62.797Mb Total transferred 157.08Mb (1.4318Mb/sec) 91.64 Requests/sec executed Test execution summary: total time: 109.7050s total number of events: 10053 total time taken by event execution: 464.7600 per-request statistics: min: 0.01ms avg: 46.23ms max: 11488.59ms approx. 95 percentile: 125.81ms Threads fairness: events (avg/stddev): 628.3125/59.81 execution time (avg/stddev): 29.0475/6.34 On the bare metal node, with the caches dropped ----------------------------------------------- After creating the 128 files, I drop the caches to get ?from SAN? results. root at nebula1:/var/lib/one/datastores/bench# sysbench --num-threads=16 --test=fileio --file-total-size=9G --file-test-mode=rndrw prepare sysbench 0.4.12: multi-threaded system evaluation benchmark 128 files, 73728Kb each, 9216Mb total Creating files for the test... # DROP CACHES root at nebula1: echo 3 > /proc/sys/vm/drop_caches root at nebula1:/var/lib/one/datastores/bench# sysbench --num-threads=16 --test=fileio --file-total-size=9G --file-test-mode=rndrw run sysbench 0.4.12: multi-threaded system evaluation benchmark Running the test with following options: Number of threads: 16 Extra file open flags: 0 128 files, 72Mb each 9Gb total file size Block size 16Kb Number of random requests for random IO: 10000 Read/Write ratio for combined random IO test: 1.50 Periodic FSYNC enabled, calling fsync() each 100 requests. Calling fsync() at the end of test, Enabled. Using synchronous I/O mode Doing random r/w test Threads started! Done. Operations performed: 6013 Read, 3999 Write, 12800 Other = 22812 Total Read 93.953Mb Written 62.484Mb Total transferred 156.44Mb (1.5465Mb/sec) 98.98 Requests/sec executed Test execution summary: total time: 101.1559s total number of events: 10012 total time taken by event execution: 1109.0862 per-request statistics: min: 0.01ms avg: 110.78ms max: 13098.27ms approx. 95 percentile: 164.52ms Threads fairness: events (avg/stddev): 625.7500/114.50 execution time (avg/stddev): 69.3179/6.54 On the bare metal node, with the test files filled in the cache --------------------------------------------------------------- I run md5sum on all the files to let the kernel cache them. # Load files in cache root at nebula1:/var/lib/one/datastores/bench# md5sum test* root at nebula1:/var/lib/one/datastores/bench# sysbench --num-threads=16 --test=fileio --file-total-size=9G --file-test-mode=rndrw run sysbench 0.4.12: multi-threaded system evaluation benchmark Running the test with following options: Number of threads: 16 Extra file open flags: 0 128 files, 72Mb each 9Gb total file size Block size 16Kb Number of random requests for random IO: 10000 Read/Write ratio for combined random IO test: 1.50 Periodic FSYNC enabled, calling fsync() each 100 requests. Calling fsync() at the end of test, Enabled. Using synchronous I/O mode Doing random r/w test Threads started! Done. Operations performed: 6069 Read, 4061 Write, 12813 Other = 22943 Total Read 94.828Mb Written 63.453Mb Total transferred 158.28Mb (54.896Mb/sec) 3513.36 Requests/sec executed Test execution summary: total time: 2.8833s total number of events: 10130 total time taken by event execution: 16.3824 per-request statistics: min: 0.01ms avg: 1.62ms max: 760.53ms approx. 95 percentile: 5.51ms Threads fairness: events (avg/stddev): 633.1250/146.90 execution time (avg/stddev): 1.0239/0.33 Footnotes: [1] https://git.samba.org/?p=ctdb.git;a=blob;f=utils/ping_pong/ping_pong.c [2] https://wiki.samba.org/index.php/Ping_pong -- Daniel Dehennin R?cup?rer ma clef GPG: gpg --recv-keys 0xCC1E9E5B7A6FE2DF Fingerprint: 3E69 014E 5C23 50E8 9ED6 2AAD CC1E 9E5B 7A6F E2DF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 342 bytes Desc: not available URL: