[Linux-cluster] DRBD Primary/Primary + GNBD + multipath + GFS with GNBD server reboot == kaboom?
Benjamin Franz
snowhare at nihongo.org
Tue Jan 22 20:25:39 UTC 2008
I configured an eight node cluster with two machines paired up with
Primary/Primary DRBD served via GNBD to the other machines with roundrobin
multipath + GFS on the client machines (the DRBD devices being served via
both DRBD machines as GNBD devices). I successfully balanced the clients
into mounting the GFS based file systems on boot but attempting to reboot
one of the DRBD pair machines ('reboot') resulted in multipath losing its
mind on all of the client machines.
It failed to transparently cut the failed GNBD server node out, instead
failing _both_ multipathed GNDB devices (which makes no particular sense
to me) and even after restoring the downed DRBD box and associated GNBD
server one client box was left so 'jammed' on the GFS file system that I
had to power cycle it to restore it (and all of the other boxes mounting
the same GFS file system) to normal operation.
This *didn't* happen when I simply powered off one of the DRBD paired
machines. Then multipath et al kept on working fine with the single GNBD
server node just like it was supposed to. Recovery went fine as well.
Among all the other messages (see attached file) I got this in
/var/log/messages on the 'jammed' GNBD client machine:
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: fatal: I/O error
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: block = 26
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: function = gfs_dreread
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: file = /builddir/build/BUILD/gfs-kmod-0.1.16/_kmod_build_/src/gfs/dio.c, line = 576
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: time = 1201024470
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: about to withdraw from the cluster
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: telling LM to withdraw
Jan 22 09:54:31 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:31 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:31 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:31 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:31 pbox7 kernel: GFS: fsid=Production0:web1.3: withdrawn
Jan 22 09:54:31 pbox7 kernel:
Jan 22 09:54:31 pbox7 kernel: Call Trace:
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883ffc94>] :gfs:gfs_lm_withdraw+0xc4/0xd3
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8001b9e6>] generic_make_request+0x1e8/0x1ff
Jan 22 09:54:31 pbox7 kernel: [<ffffffff88414dcf>] :gfs:gfs_io_error_bh_i+0x32/0x37
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883eca98>] :gfs:gfs_dreread+0xad/0xc7
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883ecada>] :gfs:gfs_dread+0x28/0x43
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883ed522>] :gfs:gfs_get_meta_buffer+0xc6/0x247
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883fa25b>] :gfs:gfs_copyin_dinode+0x1d/0x131
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883f8c78>] :gfs:inode_go_lock+0x27/0x3f
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883f6a50>] :gfs:glock_wait_internal+0x222/0x2bb
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883f6e7e>] :gfs:gfs_glock_nq+0x395/0x3d5
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883f6ed4>] :gfs:gfs_glock_nq_init+0x16/0x2a
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8840a782>] :gfs:gfs_getattr+0x41/0x67
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8000de0b>] vfs_getattr+0x2d/0xa9
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80028392>] vfs_stat_fd+0x32/0x4a
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80064a9d>] do_page_fault+0x4eb/0x81d
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80039d83>] tty_ldisc_deref+0x68/0x7b
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80094baa>] recalc_sigpending_and_wake+0x9/0x1a
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80028924>] do_sigaction+0x72/0x195
Jan 22 09:54:31 pbox7 kernel: [<ffffffff800231e6>] sys_newstat+0x19/0x31
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8005bde9>] error_exit+0x0/0x84
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8005b116>] system_call+0x7e/0x83
On a repeat experiment, here is what one of the client system looked like
just before I 'reboot'ed one of the DRBD machines:
[root at pbox7 tmp]# multipath -ll
mpath2 (GNBD-freerun-production0-drbd-web1) dm-11 GNBD,GNBD
[size=349G][features=0][hwhandler=0] \_ round-robin 0 [prio=2][active]
\_ #:#:#:# gnbd2 252:2 [active][ready]
\_ #:#:#:# gnbd5 252:5 [active][ready]
mpath1 (GNBD-freerun-production0-drbd-misc1) dm-10 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
\_ #:#:#:# gnbd1 252:1 [active][ready]
\_ #:#:#:# gnbd4 252:4 [active][ready]
mpath0 (GNBD-freerun-production0-drbd-log1) dm-9 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
\_ #:#:#:# gnbd0 252:0 [active][ready]
\_ #:#:#:# gnbd3 252:3 [active][ready]
[root at pbox7 tmp]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/pbox7-root0
48755504 6479324 39759600 15% /
/dev/sda1 194442 31032 153371 17% /boot
tmpfs 8220020 0 8220020 0% /dev/shm
/dev/mapper/ProdWeb1-data0
359748608 178031212 181717396 50% /u1
/dev/mapper/ProdMisc1-data0
359748608 152 359748456 1% /u2
/dev/mapper/ProdLog1-data0
359748608 152 359748456 1% /u3
[root at pbox7 tmp]# gnbd_import -l
Device name : log1a
----------------------
Minor # : 0
sysfs name : /block/gnbd0
Server : pbox9.internal.freerun.com
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 732399448
Device name : misc1a
----------------------
Minor # : 1
sysfs name : /block/gnbd1
Server : pbox9.internal.freerun.com
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 732399448
Device name : web1a
----------------------
Minor # : 2
sysfs name : /block/gnbd2
Server : pbox9.internal.freerun.com
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 732399448
Device name : log1b
----------------------
Minor # : 3
sysfs name : /block/gnbd3
Server : pbox10.internal.freerun.com
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 732399448
Device name : misc1b
----------------------
Minor # : 4
sysfs name : /block/gnbd4
Server : pbox10.internal.freerun.com
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 732399448
Device name : web1b
----------------------
Minor # : 5
sysfs name : /block/gnbd5
Server : pbox10.internal.freerun.com
Port : 14567
State : Open Connected Clear
Readonly : No
Sectors : 732399448
After 'reboot'ing one of the DRBD machines (while the machine was still
off):
[root at pbox7 ~]# multipath -ll
gnbd0: checker msg is "directio checker reports path is down"
gnbd1: checker msg is "directio checker reports path is down"
gnbd2: checker msg is "directio checker reports path is down"
mpath2 (GNBD-freerun-production0-drbd-web1) dm-11 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ #:#:#:# gnbd2 252:2 [failed][faulty]
\_ #:#:#:# gnbd5 252:5 [failed][faulty]
mpath1 (GNBD-freerun-production0-drbd-misc1) dm-10 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ #:#:#:# gnbd1 252:1 [failed][faulty]
\_ #:#:#:# gnbd4 252:4 [failed][faulty]
mpath0 (GNBD-freerun-production0-drbd-log1) dm-9 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ #:#:#:# gnbd0 252:0 [failed][faulty]
\_ #:#:#:# gnbd3 252:3 [failed][faulty]
And a minute later (after the DRBD/GNBD Server machine finished
rebooting):
[root at pbox7 ~]# multipath -ll
mpath2 (GNBD-freerun-production0-drbd-web1) dm-11 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
\_ #:#:#:# gnbd2 252:2 [active][ready]
\_ #:#:#:# gnbd5 252:5 [active][ready]
mpath1 (GNBD-freerun-production0-drbd-misc1) dm-10 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
\_ #:#:#:# gnbd1 252:1 [active][ready]
\_ #:#:#:# gnbd4 252:4 [active][ready]
mpath0 (GNBD-freerun-production0-drbd-log1) dm-9 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
\_ #:#:#:# gnbd0 252:0 [active][ready]
\_ #:#:#:# gnbd3 252:3 [active][ready]
Any ideas or suggestions as to what is going wrong with 'reboot'ing one of
the DRBD machines that causes multipath to lose *all* the paths?
--
Benjamin Franz
I don't think it's a good idea to follow the catalog's suggestion to force
the "Double Helix" on your biologist friends, or the "Wavelength" on your
physicist friends. They have access to virulent pathogens and liquid
nitrogen, and I just know it will end in tears before bedtime.
- Verzoeking, LiveJournal Entry, 22 Nov 2005
-------------- next part --------------
#!/bin/bash
#
# gnbd-srv Start/Stop GNBD Server
#
# chkconfig: 2345 90 09
# description: This shell script takes care of exporting and unexporting
# GNBDs devices.
# processname: gnbd_serv
# pidfile: /var/run/gnbd/gnbd_serv.pid
# source function library
. /etc/rc.d/init.d/functions
[ -r /etc/sysconfig/gnbd ] && . /etc/sysconfig/gnbd
GNBD_SERV_TIMEOUT=${GNBD_SERV_TIMEOUT:-5}
prog="GNBD Server"
start() {
if [ ! -f /etc/gnbdtab ]; then
exit 0
fi
echo -n $"Starting $prog: "
# start gnbd_serv
if [ -e /var/run/gnbd/gnbd_serv.pid ] && \
[ -e /proc/`cat /var/run/gnbd/gnbd_serv.pid` ]; then
echo -n $"gnbd_serv is already running"
failure
echo
return 1
fi
gnbd_serv -q $GNBD_SERV_OPTS
if [ $? -ne 0 ]; then
echo -n $"cannot start gnbd_serv"
failure
return 1
fi
# export all GNBDs devices
GNBD_EXPORT=`cat /etc/gnbdtab | egrep '^export'`
if [ -n "$GNBD_EXPORT" ]; then
cat /etc/gnbdtab | \
awk '/^export/ { print "-d", $3, "-e", $2, "-u", $4 }' | \
xargs -l gnbd_export $GNBD_EXPORT_OPTS > /dev/null
if [ $? -ne 0 ]; then
echo -n $"failed to export GNBDs devices"
gnbd_serv -q -K
failure
return 1
fi
fi
success
echo
return 0
}
stop() {
echo -n $"Stopping $prog: "
# check
if [ ! -e /var/run/gnbd/gnbd_serv.pid ] || \
[ ! -e /proc/`cat /var/run/gnbd/gnbd_serv.pid` ]; then
echo -n $"gnbd_serv is not running"
failure
echo
return 1
fi
# unexport all GNBDs devices
gnbd_export -q -R
if [ $? -ne 0 ]; then
echo -n $"failed to unexport all GNBDs devices"
failure
return 1
fi
# stop gnbd_serv
GNBD_SERV_PROCS="$(pgrep gnbd_serv)"
while [ -n "$GNBD_SERV_PROCS" ] && \
[ $GNBD_SERV_TIMEOUT -gt 0 ]; do
gnbd_serv -q -k
sleep 1
GNBD_SERV_TIMEOUT=$(( $GNBD_SERV_TIME -1 ))
GNBD_SERV_PROCS="$(pgrep gnbd_serv)"
done
if [ -n "$GNBD_SERV_PROCS" ]; then
gnbd_serv -q -K
fi
if [ $? -ne 0 ]; then
echo -n $"failed to stop gnbd_serv"
failure
return 1
fi
success
echo
return 0
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status gnbd_serv
;;
*)
echo $"Usage: $0 {start|stop|status}"
exit 1
;;
esac
exit $?
-------------- next part --------------
# This is an example configuration file for device mapper multipath.
# For a complete list of the default configuration values, see
# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.defaults
# For a list of configuration options with descriptions, see
# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.annotated
# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
#blacklist {
# devnode "*"
#}
## By default, devices with vendor = "IBM" and product = "S/390.*" are
## blacklisted. To enable mulitpathing on these devies, uncomment the
## following lines.
#blacklist_exceptions {
# device {
# vendor "IBM"
# product "S/390.*"
# }
#}
## Use user friendly names, instead of using WWIDs as names.
defaults {
user_friendly_names yes
}
##
## This is a template multipath-tools configuration file
## Uncomment the lines relevent to your environment
##
#defaults {
# udev_dir /dev
# polling_interval 10
# selector "round-robin 0"
# path_grouping_policy multibus
# getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
# prio_callout /bin/true
# path_checker readsector0
# rr_min_io 100
# rr_weight priorities
# failback immediate
# no_path_retry fail
# user_friendly_name yes
#}
##
## The wwid line in the following blacklist section is shown as an example
## of how to blacklist devices by wwid. The 3 devnode lines are the
## compiled in default blacklist. If you want to blacklist entire types
## of devices, such as all scsi devices, you should use a devnode line.
## However, if you want to blacklist specific devices, you should use
## a wwid line. Since there is no guarantee that a specific device will
## not change names on reboot (from /dev/sda to /dev/sdb for example)
## devnode lines are not recommended for blacklisting specific devices.
##
blacklist {
# wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda|sdb|sdc|sdd)[0-9]*"
devnode "^hd[a-z]"
}
#multipaths {
# multipath {
# wwid 3600508b4000156d700012000000b0000
# alias yellow
# path_grouping_policy multibus
# path_checker readsector0
# path_selector "round-robin 0"
# failback manual
# rr_weight priorities
# no_path_retry 5
# }
# multipath {
# wwid 1DEC_____321816758474
# alias red
# }
#}
#devices {
# device {
# vendor "COMPAQ "
# product "HSV110 (C)COMPAQ"
# path_grouping_policy multibus
# getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
# path_checker readsector0
# path_selector "round-robin 0"
# hardware_handler "0"
# failback 15
# rr_weight priorities
# no_path_retry queue
# }
# device {
# vendor "COMPAQ "
# product "MSA1000 "
# path_grouping_policy multibus
# }
#}
-------------- next part --------------
<?xml version="1.0"?>
<cluster alias="Production0" config_version="23" name="Production0">
<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="pbox7.freerun.com" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="apcpdu1" option="off" port="6"/>
<device name="apcpdu3" option="off" port="6"/>
<device name="apcpdu1" option="on" port="6"/>
<device name="apcpdu3" option="on" port="6"/>
</method>
<method name="1">
<device name="apcpdu1" option="off" port="6"/>
<device name="apcpdu3" option="off" port="6"/>
<device name="apcpdu1" option="on" port="6"/>
<device name="apcpdu3" option="on" port="6"/>
</method>
</fence>
</clusternode>
<clusternode name="pbox6.freerun.com" nodeid="3" votes="1">
<fence>
<method name="1">
<device name="apcpdu2" option="off" port="5"/>
<device name="apcpdu3" option="off" port="5"/>
<device name="apcpdu2" option="on" port="5"/>
<device name="apcpdu3" option="on" port="5"/>
</method>
<method name="1">
<device name="apcpdu2" option="off" port="5"/>
<device name="apcpdu3" option="off" port="5"/>
<device name="apcpdu2" option="on" port="5"/>
<device name="apcpdu3" option="on" port="5"/>
</method>
</fence>
</clusternode>
<clusternode name="pbox2.freerun.com" nodeid="4" votes="1">
<fence>
<method name="1">
<device name="apcpdu1" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="pbox8.freerun.com" nodeid="5" votes="1">
<fence>
<method name="1">
<device name="apcpdu1" option="off" port="7"/>
<device name="apcpdu4" option="off" port="7"/>
<device name="apcpdu1" option="on" port="7"/>
<device name="apcpdu4" option="on" port="7"/>
</method>
<method name="1">
<device name="apcpdu1" option="off" port="7"/>
<device name="apcpdu4" option="off" port="7"/>
<device name="apcpdu1" option="on" port="7"/>
<device name="apcpdu4" option="on" port="7"/>
</method>
</fence>
</clusternode>
<clusternode name="pbox5.freerun.com" nodeid="6" votes="1">
<fence>
<method name="1">
<device name="apcpdu2" option="off" port="4"/>
<device name="apcpdu4" option="off" port="4"/>
<device name="apcpdu2" option="on" port="4"/>
<device name="apcpdu4" option="on" port="4"/>
</method>
<method name="1">
<device name="apcpdu2" option="off" port="4"/>
<device name="apcpdu4" option="off" port="4"/>
<device name="apcpdu2" option="on" port="4"/>
<device name="apcpdu4" option="on" port="4"/>
</method>
</fence>
</clusternode>
<clusternode name="pbox1.freerun.com" nodeid="8" votes="1">
<fence>
<method name="1">
<device name="apcpdu4" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="pbox9.freerun.com" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="apcpdu2" option="off" port="2"/>
<device name="apcpdu3" option="off" port="2"/>
<device name="apcpdu4" option="off" port="2"/>
<device name="apcpdu2" option="on" port="2"/>
<device name="apcpdu3" option="on" port="2"/>
<device name="apcpdu4" option="on" port="2"/>
</method>
<method name="1">
<device name="apcpdu2" option="off" port="2"/>
<device name="apcpdu3" option="off" port="2"/>
<device name="apcpdu4" option="off" port="2"/>
<device name="apcpdu2" option="on" port="2"/>
<device name="apcpdu3" option="on" port="2"/>
<device name="apcpdu4" option="on" port="2"/>
</method>
<method name="1">
<device name="apcpdu2" option="off" port="2"/>
<device name="apcpdu3" option="off" port="2"/>
<device name="apcpdu4" option="off" port="2"/>
<device name="apcpdu2" option="on" port="2"/>
<device name="apcpdu3" option="on" port="2"/>
<device name="apcpdu4" option="on" port="2"/>
</method>
</fence>
</clusternode>
<clusternode name="pbox10.freerun.com" nodeid="7" votes="1">
<fence>
<method name="1">
<device name="apcpdu1" option="off" port="3"/>
<device name="apcpdu2" option="off" port="3"/>
<device name="apcpdu3" option="off" port="3"/>
<device name="apcpdu1" option="on" port="3"/>
<device name="apcpdu2" option="on" port="3"/>
<device name="apcpdu3" option="on" port="3"/>
</method>
<method name="1">
<device name="apcpdu1" option="off" port="3"/>
<device name="apcpdu2" option="off" port="3"/>
<device name="apcpdu3" option="off" port="3"/>
<device name="apcpdu1" option="on" port="3"/>
<device name="apcpdu2" option="on" port="3"/>
<device name="apcpdu3" option="on" port="3"/>
</method>
<method name="1">
<device name="apcpdu1" option="off" port="3"/>
<device name="apcpdu2" option="off" port="3"/>
<device name="apcpdu3" option="off" port="3"/>
<device name="apcpdu1" option="on" port="3"/>
<device name="apcpdu2" option="on" port="3"/>
<device name="apcpdu3" option="on" port="3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apcpdu1.freerun.com" login="device" name="apcpdu1" passwd="apassword"/>
<fencedevice agent="fence_apc" ipaddr="apcpdu2.freerun.com" login="device" name="apcpdu2" passwd="apassword"/>
<fencedevice agent="fence_apc" ipaddr="apcpdu3.freerun.com" login="device" name="apcpdu3" passwd="apassword"/>
<fencedevice agent="fence_apc" ipaddr="apcpdu4.freerun.com" login="device" name="apcpdu4" passwd="apassword"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="LinuxVirtualServer" ordered="1" restricted="1">
<failoverdomainnode name="pbox7.freerun.com" priority="3"/>
<failoverdomainnode name="pbox6.freerun.com" priority="2"/>
<failoverdomainnode name="pbox8.freerun.com" priority="4"/>
<failoverdomainnode name="pbox5.freerun.com" priority="1"/>
<failoverdomainnode name="pbox9.freerun.com" priority="5"/>
<failoverdomainnode name="pbox10.freerun.com" priority="6"/>
</failoverdomain>
<failoverdomain name="SonicGateway" ordered="1" restricted="1">
<failoverdomainnode name="pbox2.freerun.com" priority="2"/>
<failoverdomainnode name="pbox1.freerun.com" priority="1"/>
</failoverdomain>
</failoverdomains>
<resources/>
</rm>
</cluster>
-------------- next part --------------
Jan 22 09:52:07 pbox7 clurgmgrd[5892]: <notice> Member 2 shutting down
Jan 22 09:52:23 pbox7 kernel: gnbd0: Receive control failed (result -32)
Jan 22 09:52:23 pbox7 kernel: gnbd0: shutting down socket
Jan 22 09:52:23 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:52:23 pbox7 gnbd_recvd[4974]: client lost connection with pbox9.internal.freerun.com : Broken pipe
Jan 22 09:52:23 pbox7 gnbd_recvd[4974]: reconnecting
Jan 22 09:52:23 pbox7 kernel: gnbd2: Receive control failed (result -32)
Jan 22 09:52:23 pbox7 kernel: gnbd2: shutting down socket
Jan 22 09:52:23 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:52:23 pbox7 gnbd_recvd[4994]: client lost connection with pbox9.internal.freerun.com : Broken pipe
Jan 22 09:52:23 pbox7 gnbd_recvd[4994]: reconnecting
Jan 22 09:52:23 pbox7 kernel: gnbd1: Receive control failed (result -32)
Jan 22 09:52:23 pbox7 kernel: gnbd1: shutting down socket
Jan 22 09:52:23 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:52:23 pbox7 gnbd_recvd[4984]: client lost connection with pbox9.internal.freerun.com : Broken pipe
Jan 22 09:52:23 pbox7 gnbd_recvd[4984]: reconnecting
Jan 22 09:52:28 pbox7 gnbd_recvd[4974]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused
Jan 22 09:52:28 pbox7 gnbd_recvd[4974]: reconnecting
Jan 22 09:52:28 pbox7 gnbd_recvd[4994]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused
Jan 22 09:52:28 pbox7 gnbd_recvd[4994]: reconnecting
Jan 22 09:52:28 pbox7 gnbd_recvd[4984]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused
Jan 22 09:52:28 pbox7 gnbd_recvd[4984]: reconnecting
Jan 22 09:52:30 pbox7 kernel: gnbd0: Attempted send on closed socket
Jan 22 09:52:32 pbox7 kernel: device-mapper: multipath: Failing path 252:0.
Jan 22 09:52:32 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:52:32 pbox7 multipathd: checker failed path 252:0 in map mpath0
Jan 22 09:52:32 pbox7 multipathd: mpath0: remaining active paths: 1
Jan 22 09:52:33 pbox7 kernel: gnbd1: Attempted send on closed socket
Jan 22 09:52:33 pbox7 gnbd_recvd[4974]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused
Jan 22 09:52:33 pbox7 gnbd_recvd[4974]: reconnecting
Jan 22 09:52:33 pbox7 gnbd_recvd[4994]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused
Jan 22 09:52:33 pbox7 gnbd_recvd[4994]: reconnecting
Jan 22 09:52:33 pbox7 gnbd_recvd[4984]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused
Jan 22 09:52:33 pbox7 gnbd_recvd[4984]: reconnecting
Jan 22 09:52:35 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:52:35 pbox7 kernel: device-mapper: multipath: Failing path 252:1.
Jan 22 09:52:35 pbox7 kernel: gnbd2: Attempted send on closed socket
Jan 22 09:52:35 pbox7 multipathd: checker failed path 252:1 in map mpath1
Jan 22 09:52:35 pbox7 multipathd: mpath1: remaining active paths: 1
Jan 22 09:52:37 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:52:37 pbox7 multipathd: checker failed path 252:2 in map mpath2
Jan 22 09:52:37 pbox7 kernel: device-mapper: multipath: Failing path 252:2.
Jan 22 09:52:38 pbox7 openais[3472]: [TOTEM] entering GATHER state from 12.
Jan 22 09:52:43 pbox7 multipathd: mpath2: remaining active paths: 1
Jan 22 09:52:43 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] entering GATHER state from 0.
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] Saving state aru eb high seq received eb
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] Storing new sequence id for ring 50c
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] entering COMMIT state.
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] entering RECOVERY state.
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [0] member 64.142.17.1:
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [1] member 64.142.17.2:
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [2] member 64.142.17.5:
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [3] member 64.142.17.6:
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [4] member 64.142.17.7:
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [5] member 64.142.17.8:
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [6] member 64.142.17.10:
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] Did not need to originate any messages in recovery.
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] CLM CONFIGURATION CHANGE
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] New Configuration:
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.1)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.2)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.5)
Jan 22 09:52:43 pbox7 kernel: dlm: closing connection to node 2
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.6)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.7)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.8)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.10)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] Members Left:
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.9)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] Members Joined:
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] CLM CONFIGURATION CHANGE
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] New Configuration:
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.1)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.2)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.5)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.6)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.7)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.8)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] r(0) ip(64.142.17.10)
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] Members Left:
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] Members Joined:
Jan 22 09:52:43 pbox7 openais[3472]: [SYNC ] This node is within the primary component and will provide service.
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] entering OPERATIONAL state.
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] got nodejoin message 64.142.17.1
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] got nodejoin message 64.142.17.2
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] got nodejoin message 64.142.17.5
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] got nodejoin message 64.142.17.6
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] got nodejoin message 64.142.17.7
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] got nodejoin message 64.142.17.8
Jan 22 09:52:43 pbox7 openais[3472]: [CLM ] got nodejoin message 64.142.17.10
Jan 22 09:52:46 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:52:48 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:52:48 pbox7 kernel: gnbd_monitor 11430 called gnbd_end_request with an error
Jan 22 09:52:48 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:52:48 pbox7 kernel: gnbd_monitor 11431 called gnbd_end_request with an error
Jan 22 09:52:48 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:52:48 pbox7 kernel: gnbd_monitor 11432 called gnbd_end_request with an error
Jan 22 09:52:48 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:52:51 pbox7 openais[3472]: [CPG ] got joinlist message from node 7
Jan 22 09:52:51 pbox7 openais[3472]: [CPG ] got joinlist message from node 8
Jan 22 09:52:51 pbox7 openais[3472]: [CPG ] got joinlist message from node 4
Jan 22 09:52:51 pbox7 openais[3472]: [CPG ] got joinlist message from node 6
Jan 22 09:52:51 pbox7 openais[3472]: [CPG ] got joinlist message from node 3
Jan 22 09:52:51 pbox7 openais[3472]: [CPG ] got joinlist message from node 1
Jan 22 09:52:51 pbox7 openais[3472]: [CPG ] got joinlist message from node 5
Jan 22 09:52:52 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:52:53 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:52:53 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:52:57 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:52:57 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:52:57 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:52:58 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:52:58 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:52:58 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:52:58 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:52:58 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:53:02 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:02 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:02 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:53:02 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:53:04 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:53:04 pbox7 kernel: device-mapper: multipath: Failing path 252:3.
Jan 22 09:53:04 pbox7 multipathd: checker failed path 252:3 in map mpath0
Jan 22 09:53:04 pbox7 multipathd: mpath0: remaining active paths: 0
Jan 22 09:53:05 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:05 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:05 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:53:05 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:05 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:07 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:53:07 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:53:07 pbox7 kernel: device-mapper: multipath: Failing path 252:5.
Jan 22 09:53:07 pbox7 multipathd: checker failed path 252:5 in map mpath2
Jan 22 09:53:07 pbox7 multipathd: mpath2: remaining active paths: 0
Jan 22 09:53:11 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:11 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:11 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:53:13 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:53:14 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:14 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:14 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:53:14 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:14 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:16 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:53:16 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:53:20 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:20 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:20 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:53:22 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:53:23 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:23 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:23 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:23 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:23 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:53:25 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:53:25 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:53:29 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:29 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:29 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:53:31 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:53:32 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:32 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:32 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:53:32 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:32 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:34 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:53:34 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:53:38 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:38 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:38 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:53:40 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:53:40 pbox7 gnbd_recvd[11488]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:53:40 pbox7 gnbd_recvd[11488]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:53:40 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:53:41 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:41 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:41 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:53:41 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:41 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:43 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:53:43 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:53:43 pbox7 gnbd_recvd[11504]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:53:43 pbox7 gnbd_recvd[11504]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:53:43 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:53:46 pbox7 gnbd_recvd[11513]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:53:46 pbox7 gnbd_recvd[11513]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:53:46 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:53:47 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:47 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:47 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:53:49 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:53:50 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:50 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:50 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:53:50 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:50 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:52 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:53:52 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:53:56 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:56 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:56 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:53:58 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:53:59 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:59 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:59 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:53:59 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:59 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:01 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:54:01 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:54:05 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:05 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:05 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:54:06 pbox7 kernel: gnbd (pid 5029: gnbd_recvd) got signal
Jan 22 09:54:06 pbox7 kernel: gnbd5: Receive control failed (result -4)
Jan 22 09:54:06 pbox7 kernel: gnbd5: shutting down socket
Jan 22 09:54:06 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:54:06 pbox7 gnbd_recvd[5029]: client lost connection with pbox10.internal.freerun.com : Interrupted system call
Jan 22 09:54:06 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:06 pbox7 kernel: gnbd (pid 5007: gnbd_recvd) got signal
Jan 22 09:54:06 pbox7 kernel: gnbd3: Receive control failed (result -4)
Jan 22 09:54:06 pbox7 kernel: gnbd3: shutting down socket
Jan 22 09:54:06 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:54:06 pbox7 gnbd_recvd[5007]: client lost connection with pbox10.internal.freerun.com : Interrupted system call
Jan 22 09:54:06 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:07 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:54:08 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:08 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:08 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:08 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:08 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:54:10 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:54:10 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:54:11 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:11 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:11 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:11 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:14 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:14 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:14 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:54:16 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:54:16 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:16 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:16 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:16 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:17 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:17 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:17 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:54:17 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:17 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:19 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:54:19 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:54:21 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:21 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:21 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:21 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:23 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:23 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:23 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:54:25 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:54:26 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:26 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:54:26 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:26 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:26 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:26 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:26 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:26 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:26 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:28 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:54:28 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: fatal: I/O error
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: block = 26
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: function = gfs_dreread
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: file = /builddir/build/BUILD/gfs-kmod-0.1.16/_kmod_build_/src/gfs/dio.c, line = 576
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: time = 1201024470
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: about to withdraw from the cluster
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: telling LM to withdraw
Jan 22 09:54:31 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:31 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:31 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:31 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:31 pbox7 kernel: GFS: fsid=Production0:web1.3: withdrawn
Jan 22 09:54:31 pbox7 kernel:
Jan 22 09:54:31 pbox7 kernel: Call Trace:
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883ffc94>] :gfs:gfs_lm_withdraw+0xc4/0xd3
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8001b9e6>] generic_make_request+0x1e8/0x1ff
Jan 22 09:54:31 pbox7 kernel: [<ffffffff88414dcf>] :gfs:gfs_io_error_bh_i+0x32/0x37
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883eca98>] :gfs:gfs_dreread+0xad/0xc7
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883ecada>] :gfs:gfs_dread+0x28/0x43
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883ed522>] :gfs:gfs_get_meta_buffer+0xc6/0x247
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883fa25b>] :gfs:gfs_copyin_dinode+0x1d/0x131
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883f8c78>] :gfs:inode_go_lock+0x27/0x3f
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883f6a50>] :gfs:glock_wait_internal+0x222/0x2bb
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883f6e7e>] :gfs:gfs_glock_nq+0x395/0x3d5
Jan 22 09:54:31 pbox7 kernel: [<ffffffff883f6ed4>] :gfs:gfs_glock_nq_init+0x16/0x2a
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8840a782>] :gfs:gfs_getattr+0x41/0x67
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8000de0b>] vfs_getattr+0x2d/0xa9
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80028392>] vfs_stat_fd+0x32/0x4a
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80064a9d>] do_page_fault+0x4eb/0x81d
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80039d83>] tty_ldisc_deref+0x68/0x7b
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80094baa>] recalc_sigpending_and_wake+0x9/0x1a
Jan 22 09:54:31 pbox7 kernel: [<ffffffff80028924>] do_sigaction+0x72/0x195
Jan 22 09:54:31 pbox7 kernel: [<ffffffff800231e6>] sys_newstat+0x19/0x31
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8005bde9>] error_exit+0x0/0x84
Jan 22 09:54:31 pbox7 kernel: [<ffffffff8005b116>] system_call+0x7e/0x83
Jan 22 09:54:31 pbox7 kernel:
Jan 22 09:54:32 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:32 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:32 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:54:34 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:54:35 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:35 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:35 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:54:35 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:35 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:36 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:36 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:36 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:36 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:37 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:54:37 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:54:39 pbox7 gnbd_recvd[11632]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:54:39 pbox7 gnbd_recvd[11632]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:54:39 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:54:41 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:41 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:41 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:54:42 pbox7 gnbd_recvd[11673]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:54:42 pbox7 gnbd_recvd[11673]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:54:42 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:54:43 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:54:44 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:44 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:44 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:54:44 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:44 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:45 pbox7 gnbd_recvd[11678]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:54:45 pbox7 gnbd_recvd[11678]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:54:45 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:54:46 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:54:46 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:54:50 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:50 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:50 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:54:50 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:50 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:50 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:50 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:52 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:54:53 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:53 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:53 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:53 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:53 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:54:55 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:54:55 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:54:55 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:55 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:55 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:55 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:59 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:59 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:59 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:55:00 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:00 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:00 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:00 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:01 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:55:02 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:02 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:55:02 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:55:02 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:02 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:55:04 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:55:04 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:55:05 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:05 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:05 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:05 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:08 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:08 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:55:08 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:55:10 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:55:10 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:10 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:10 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:10 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:11 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:11 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:55:11 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:55:11 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:11 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:55:13 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:55:13 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:55:15 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:15 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:15 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:15 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:17 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:17 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:55:17 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:55:19 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:55:20 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:20 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:55:20 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:55:20 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:20 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:55:20 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:20 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:20 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:20 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:22 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:55:22 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:55:25 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:25 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:25 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:25 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:26 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:26 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:55:26 pbox7 multipathd: gnbd0: directio checker reports path is down
Jan 22 09:55:28 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:55:29 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:29 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:55:29 pbox7 multipathd: gnbd1: directio checker reports path is down
Jan 22 09:55:29 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:29 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:55:30 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:30 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:30 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:30 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:31 pbox7 multipathd: gnbd2: directio checker reports path is down
Jan 22 09:55:31 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:55:31 pbox7 multipathd: mpath0: stop event checker thread
Jan 22 09:55:35 pbox7 kernel: gnbd3: Attempted send on closed socket
Jan 22 09:55:35 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:35 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:35 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:35 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:37 pbox7 multipathd: gnbd3: checker msg is "directio checker reports path is down"
Jan 22 09:55:37 pbox7 kernel: gnbd5: Attempted send on closed socket
Jan 22 09:55:38 pbox7 gnbd_recvd[11776]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:55:38 pbox7 gnbd_recvd[11776]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:55:38 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:55:39 pbox7 multipathd: gnbd5: checker msg is "directio checker reports path is down"
Jan 22 09:55:41 pbox7 multipathd: gnbd3: checker msg is "directio checker reports path is down"
Jan 22 09:55:41 pbox7 gnbd_recvd[11788]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:55:41 pbox7 gnbd_recvd[11788]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:55:41 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:55:43 pbox7 multipathd: gnbd5: checker msg is "directio checker reports path is down"
Jan 22 09:55:43 pbox7 multipathd: mpath0: event checker started
Jan 22 09:55:43 pbox7 multipathd: mpath1: event checker started
Jan 22 09:55:43 pbox7 multipathd: mpath2: event checker started
Jan 22 09:55:43 pbox7 multipathd: path checkers start up
Jan 22 09:55:43 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:55:43 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:55:43 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:55:44 pbox7 gnbd_recvd[11793]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:55:44 pbox7 gnbd_recvd[11793]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:55:44 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:55:45 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:55:47 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:55:49 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:49 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:49 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:49 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:52 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:55:54 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:55:54 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:55:54 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:55:54 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:54 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:54 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:54 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:55:56 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:55:59 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:59 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:55:59 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:55:59 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:01 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:56:03 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:56:03 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:56:03 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:56:04 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:04 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:04 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:04 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:05 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:56:09 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:09 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:09 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:09 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:10 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:56:12 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:56:12 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:56:12 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:56:14 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:56:14 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:14 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:14 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:14 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:15 pbox7 kernel: gnbd3: Attempted send on closed socket
Jan 22 09:56:17 pbox7 kernel: gnbd5: Attempted send on closed socket
Jan 22 09:56:19 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:56:19 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:19 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:19 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:19 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:21 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:56:21 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:56:21 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:56:23 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:56:23 pbox7 multipathd: dm-10: add map (uevent)
Jan 22 09:56:23 pbox7 multipathd: dm-10: devmap already registered
Jan 22 09:56:24 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:24 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:24 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:24 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:28 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:56:29 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:29 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:29 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:29 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:30 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:56:30 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:56:30 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:56:32 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:56:34 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:34 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:34 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:34 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:37 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:56:37 pbox7 gnbd_recvd[11891]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:56:37 pbox7 gnbd_recvd[11891]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:56:37 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:56:39 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:56:39 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:56:39 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:56:40 pbox7 gnbd_recvd[11894]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:56:40 pbox7 gnbd_recvd[11894]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:56:40 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:56:41 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:56:43 pbox7 gnbd_recvd[11904]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:56:43 pbox7 gnbd_recvd[11904]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:56:43 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:56:46 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:56:48 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:56:48 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:56:48 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:56:48 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:48 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:48 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:48 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:50 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:56:53 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:53 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:53 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:53 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:55 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:56:57 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:56:57 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:56:57 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:56:58 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:58 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:56:58 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:56:58 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:56:59 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:57:03 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:03 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:03 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:03 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:04 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:57:06 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:57:06 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:57:06 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:57:08 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:57:08 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:08 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:08 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:08 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:13 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:57:13 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:13 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:13 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:13 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:15 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:57:15 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:57:15 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:57:17 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:57:18 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:18 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:18 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:18 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:22 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:57:23 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:23 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:23 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:23 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:24 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:57:24 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:57:24 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:57:26 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:57:28 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:28 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:28 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:28 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:31 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:57:33 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:57:33 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:57:33 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:57:33 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:33 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:33 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:33 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:35 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:57:36 pbox7 gnbd_recvd[11973]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:57:36 pbox7 gnbd_recvd[11973]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:57:36 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:57:39 pbox7 gnbd_recvd[11983]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:57:39 pbox7 gnbd_recvd[11983]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:57:39 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:57:40 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:57:40 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:57:40 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:57:42 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:57:42 pbox7 gnbd_recvd[11993]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:57:42 pbox7 gnbd_recvd[11993]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:57:42 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:57:44 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:57:47 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:47 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:47 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:47 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:49 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:57:51 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:57:51 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:57:51 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:57:52 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:52 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:52 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:52 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:53 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:57:57 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:57 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:57:57 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:57:57 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:57:58 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:58:00 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:58:00 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:58:00 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:58:02 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:58:02 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:02 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:02 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:02 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:07 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:58:07 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:07 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:07 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:07 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:09 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:58:09 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:58:09 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:58:11 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:58:12 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:12 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:12 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:12 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:16 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:58:17 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:17 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:17 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:17 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:18 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:58:18 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:58:18 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:58:20 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:58:22 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:22 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:22 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:22 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:25 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:58:27 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:58:27 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:58:27 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:58:27 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:27 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:27 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:27 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:29 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:58:32 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:32 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:32 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:32 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:34 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:58:35 pbox7 gnbd_recvd[12134]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:58:35 pbox7 gnbd_recvd[12134]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:58:35 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:58:36 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:58:36 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:58:36 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:58:38 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:58:38 pbox7 gnbd_recvd[12151]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:58:38 pbox7 gnbd_recvd[12151]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:58:38 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:58:41 pbox7 gnbd_recvd[12155]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:58:41 pbox7 gnbd_recvd[12155]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:58:41 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:58:43 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:58:43 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:58:43 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:58:45 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:58:46 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:46 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:46 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:46 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:47 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:58:51 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:51 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:51 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:51 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:58:52 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:58:54 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:58:54 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:58:54 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:58:56 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:58:56 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:56 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:58:56 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:58:56 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:01 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:59:01 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:01 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:01 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:01 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:03 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:59:03 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:59:03 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:59:05 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:59:06 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:06 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:06 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:06 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:10 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:59:11 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:11 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:11 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:11 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:12 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:59:12 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:59:12 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:59:14 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:59:16 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:16 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:16 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:16 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:19 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:59:21 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:59:21 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:59:21 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:59:21 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:21 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:21 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:21 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:23 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:59:26 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:26 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:26 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:26 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:28 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:59:30 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:59:30 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:59:30 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:59:31 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:31 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:31 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:31 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:32 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:59:34 pbox7 gnbd_recvd[12237]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:59:34 pbox7 gnbd_recvd[12237]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:59:34 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:59:37 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:59:37 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:59:37 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:59:37 pbox7 gnbd_recvd[12246]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:59:37 pbox7 gnbd_recvd[12246]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:59:37 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:59:39 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:59:40 pbox7 gnbd_recvd[12249]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 09:59:40 pbox7 gnbd_recvd[12249]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 09:59:40 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 09:59:41 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:59:45 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:45 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:45 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:45 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:46 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:59:48 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:59:48 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:59:48 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:59:50 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 09:59:50 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:50 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:50 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:50 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:55 pbox7 multipathd: gnbd0: unusable path
Jan 22 09:59:55 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:55 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:59:55 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:59:55 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:59:57 pbox7 multipathd: gnbd1: unusable path
Jan 22 09:59:57 pbox7 multipathd: gnbd2: unusable path
Jan 22 09:59:57 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 09:59:59 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:00:00 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:00 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:00 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:00 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:04 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:00:05 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:05 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:05 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:05 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:06 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:00:06 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:00:06 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:00:08 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:00:10 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:10 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:10 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:10 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:13 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:00:15 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:00:15 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:00:15 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:00:15 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:15 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:15 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:15 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:17 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:00:20 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:20 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:20 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:20 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:22 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:00:24 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:00:24 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:00:24 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:00:25 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:25 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:25 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:25 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:26 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:00:30 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:30 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:30 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:30 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:31 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:00:33 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:00:33 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:00:33 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:00:33 pbox7 gnbd_recvd[12330]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 10:00:33 pbox7 gnbd_recvd[12330]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 10:00:33 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 10:00:35 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:00:36 pbox7 gnbd_recvd[12333]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 10:00:36 pbox7 gnbd_recvd[12333]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 10:00:36 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 10:00:39 pbox7 gnbd_recvd[12342]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 10:00:39 pbox7 gnbd_recvd[12342]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 10:00:39 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 10:00:40 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:00:40 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:00:40 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:00:42 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:00:44 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:00:44 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:44 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:44 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:44 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:49 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:00:49 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:49 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:49 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:49 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:51 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:00:51 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:00:51 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:00:53 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:00:54 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:54 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:54 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:54 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:00:58 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:00:59 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:59 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:00:59 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:00:59 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:00 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:01:00 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:01:00 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:01:02 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:01:04 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:04 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:04 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:04 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:07 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:01:09 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:01:09 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:01:09 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:01:09 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:09 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:09 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:09 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:11 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:01:14 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:14 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:14 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:14 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:16 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:01:18 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:01:18 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:01:18 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:01:19 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:19 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:19 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:19 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:20 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:01:24 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:24 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:24 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:24 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:24 pbox7 kernel: gnbd0: GNBD_DISCONNECT
Jan 22 10:01:25 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:01:27 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:01:27 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:01:27 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:01:29 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:01:29 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:29 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:29 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:29 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:31 pbox7 gnbd_recvd[12452]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 10:01:31 pbox7 gnbd_recvd[12452]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 10:01:31 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 10:01:34 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:01:34 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:01:34 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:01:34 pbox7 gnbd_recvd[12456]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 10:01:34 pbox7 gnbd_recvd[12456]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 10:01:34 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 10:01:36 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:01:37 pbox7 gnbd_recvd[12466]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host
Jan 22 10:01:37 pbox7 gnbd_recvd[12466]: ERROR [gnbd_recvd.c:390] quitting
Jan 22 10:01:37 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1)
Jan 22 10:01:38 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:01:42 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:42 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:42 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:42 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:43 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:01:45 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:01:45 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:01:45 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:01:47 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:01:47 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:47 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:47 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:47 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:52 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:01:52 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:52 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:52 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:52 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:01:54 pbox7 multipathd: gnbd1: unusable path
Jan 22 10:01:54 pbox7 multipathd: gnbd2: unusable path
Jan 22 10:01:54 pbox7 multipathd: gnbd3: directio checker reports path is down
Jan 22 10:01:56 pbox7 multipathd: gnbd5: directio checker reports path is down
Jan 22 10:01:57 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:57 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:01:57 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:01:57 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 10:02:01 pbox7 multipathd: gnbd0: unusable path
Jan 22 10:02:02 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 10:02:02 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 10:02:02 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
More information about the Linux-cluster
mailing list