From gianluca.cecchi at gmail.com Tue Jul 26 14:00:41 2016 From: gianluca.cecchi at gmail.com (Gianluca Cecchi) Date: Tue, 26 Jul 2016 16:00:41 +0200 Subject: [rhelv6-list] Limiting pvmove speed with control groups doesn't work as expected Message-ID: Hello, I'm testing what in subject because I have to move from a SAN storage to another one and I want to limit impact on source storage. System is RH EL 6.5 and source and target of pvmove are two multipath devices Basic steps done: $ sudo yum install libcgroup $ sudo service cgconfig start Starting cgconfig service: [ OK ] $ sudo service cgconfig status Running Source multipath device has 8 paths, divided in two groups with 4 path active: $ sudo multipath -l 360a9800037543543592442595559337a 360a9800037543543592442595559337a dm-2 NETAPP,LUN size=50G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='round-robin 0' prio=0 status=active | |- 7:0:2:13 sdah 66:16 active undef running | |- 7:0:3:13 sdaw 67:0 active undef running | |- 8:0:2:13 sdcp 69:208 active undef running | `- 8:0:3:13 sdde 70:192 active undef running `-+- policy='round-robin 0' prio=0 status=enabled |- 7:0:0:13 sdd 8:48 active undef running |- 7:0:1:13 sds 65:32 active undef running |- 8:0:0:13 sdbl 67:240 active undef running `- 8:0:1:13 sdca 68:224 active undef running $ ll /dev/dm-2 brw-rw---- 1 root disk 253, 2 Jul 25 13:30 /dev/dm-2 $ sudo cgcreate -g blkio:/30M # echo "253:2 31457280" > /cgroup/blkio/30M/blkio.throttle.read_bps_device the multipath device is a PV of a VG $ sudo pvs /dev/mapper/360a9800037543543592442595559337a PV VG Fmt Attr PSize PFree /dev/mapper/360a9800037543543592442595559337a VG_ALMTEST_DATA lvm2 a-- 50.00g 0 Without limits, if I execute this command dd if=/dev/mapper/360a9800037543543592442595559337a of=/dev/null bs=1024k count=10240 I get iostat -d 3 -m -p /dev/sdah,/dev/sdaw,/dev/sdcp,/dev/sdde Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdah 333.67 40.81 0.00 122 0 sdaw 333.67 40.76 0.00 122 0 sdcp 333.67 41.31 0.00 123 0 sdde 334.00 41.04 0.00 123 0 Instead called with cgexec: # cgexec -g blkio:30M time dd if=/dev/mapper/360a9800037543543592442595559337a of=/dev/null bs=1024k count=10240 I get Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdah 64.00 7.51 0.00 22 0 sdaw 64.00 7.60 0.00 22 0 sdcp 64.00 7.43 0.00 22 0 sdde 64.33 7.45 0.00 22 0 So far so good. I have my 30MB/s... But then if I try to use the same approach with pvmove command: # cgexec -g blkio:30M pvmove -i 60 /dev/mapper/360a9800037543543592442595559337a /dev/mapper/3600a098038303769752b495147377857 I get Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdah 29.00 14.50 0.00 43 0 sdaw 29.00 14.37 0.00 43 0 sdcp 28.67 14.29 0.00 42 0 sdde 28.67 14.33 0.00 43 0 it seems an intermediate .... The target multipath device is of this kind: $ sudo multipath -l /dev/mapper/3600a098038303769752b495147377857 3600a098038303769752b495147377857 dm-39 NETAPP,LUN C-Mode size=50G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 7:0:5:13 sdds 71:160 active undef running `- 8:0:5:13 sdee 128:96 active undef running I see also confirmed in terms of time elapsed (about 15 minutes for 50Gb pvmove) that aggregate bandwith has been indeed 60MB/s instead of 30MB/s And also from iotop command where I see that apparently the main process accounting for I/O is named kcopyd using 60MB/s in reads.... Tapped also to 10MB/s with another control group but the real bandwith during pvmove remains around 60 MB/s Made the policy also for the single active paths with same results. Thanks in advance for any insight or suggestions Gianluca -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianluca.cecchi at gmail.com Tue Jul 26 14:26:01 2016 From: gianluca.cecchi at gmail.com (Gianluca Cecchi) Date: Tue, 26 Jul 2016 16:26:01 +0200 Subject: [rhelv6-list] Limiting pvmove speed with control groups doesn't work as expected In-Reply-To: References: Message-ID: On Tue, Jul 26, 2016 at 4:00 PM, Gianluca Cecchi wrote: > > > But then if I try to use the same approach with pvmove command: > > # cgexec -g blkio:30M pvmove -i 60 > /dev/mapper/360a9800037543543592442595559337a /dev/mapper/3600a098038303769752b495147377857 > > I get > > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn > sdah 29.00 14.50 0.00 43 0 > sdaw 29.00 14.37 0.00 43 0 > sdcp 28.67 14.29 0.00 42 0 > sdde 28.67 14.33 0.00 43 0 > > it seems an intermediate .... > > BTW: I also tested using "--sticky" option # cgexec -g blkio:30M --sticky pvmove . . . . but same behavior. Indeed I don't see any explicit child process originated by the pvmove command.... Gianluca -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at brimer.org Tue Jul 26 14:47:01 2016 From: lists at brimer.org (Barry Brimer) Date: Tue, 26 Jul 2016 09:47:01 -0500 Subject: [rhelv6-list] Limiting pvmove speed with control groups doesn't work as expected In-Reply-To: References: Message-ID: Maybe ionice could help? On July 26, 2016 9:26:01 AM CDT, Gianluca Cecchi wrote: >On Tue, Jul 26, 2016 at 4:00 PM, Gianluca Cecchi wrote: > >> >> >> But then if I try to use the same approach with pvmove command: >> >> # cgexec -g blkio:30M pvmove -i 60 >> /dev/mapper/360a9800037543543592442595559337a >/dev/mapper/3600a098038303769752b495147377857 >> >> I get >> >> Device: tps MB_read/s MB_wrtn/s MB_read >MB_wrtn >> sdah 29.00 14.50 0.00 43 >0 >> sdaw 29.00 14.37 0.00 43 >0 >> sdcp 28.67 14.29 0.00 42 >0 >> sdde 28.67 14.33 0.00 43 >0 >> >> it seems an intermediate .... >> >> >BTW: I also tested using "--sticky" option > ># cgexec -g blkio:30M --sticky pvmove . . . . > >but same behavior. Indeed I don't see any explicit child process >originated >by the pvmove command.... > >Gianluca > > >------------------------------------------------------------------------ > >_______________________________________________ >rhelv6-list mailing list >rhelv6-list at redhat.com >https://www.redhat.com/mailman/listinfo/rhelv6-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianluca.cecchi at gmail.com Tue Jul 26 15:51:15 2016 From: gianluca.cecchi at gmail.com (Gianluca Cecchi) Date: Tue, 26 Jul 2016 17:51:15 +0200 Subject: [rhelv6-list] Limiting pvmove speed with control groups doesn't work as expected In-Reply-To: References: Message-ID: On Tue, Jul 26, 2016 at 4:47 PM, Barry Brimer wrote: > Maybe ionice could help? > > Thanks for your answer Barry, but It would be suboptimal in my opinion for my target that is limiting read overall bandwith.... But I found it !!! With iotop I see the PID of kcopyd that starts right after pvmove command.... (the kcopyd pid is different every pvmove run, not a fixed process) [root at dbatest1 ~]# ps -ef|grep 11139 root 11139 2 1 17:40 ? 00:00:02 [kcopyd] root 17103 19305 0 17:43 pts/0 00:00:00 grep 11139 BTW, kcopyd is child of kthreadd: root 2 0 0 Jun29 ? 00:00:00 [kthreadd] So I put kcopyd process into the control group too # echo 11139 > /cgroup/blkio/30M/tasks so that I now have pvmove process and kcopyd process [root at dbatest1 ~]# cat /cgroup/blkio/30M/tasks 11139 12765 And in fact: Total DISK READ: 30399.94 K/s | Total DISK WRITE: 30345.65 K/s PID PRIO USER DISK READ> DISK WRITE SWAPIN IO COMMAND 11139 be/4 root 29970.89 K/s 0.00 K/s 0.00 % 0.00 % [kcopyd] 12765 be/4 root 65.40 K/s 0.00 K/s 0.00 % 0.91 % pvmove -i 20 /dev/mapper/360a9800037543543~v/mapper/3600a098038303769752b495147377868 36462 be/4 oracle11 20.93 K/s 15.70 K/s 0.00 % 0.08 % ora_ckpt_ALMTEST 8953 ?dif root 7.85 K/s 0.49 K/s 0.00 % 0.12 % qdiskd 20972 be/4 root 0.00 K/s 0.00 K/s 0.00 % 0.05 % [kjournald] 15733 be/4 root 0.00 K/s 1.31 K/s 0.00 % 0.00 % dsm_om_connsvcd -run 14644 be/4 root 0.00 K/s 1.31 K/s 0.00 % 0.00 % nbdisco 40736 be/4 oracle11 0.00 K/s 15.70 K/s 0.00 % 0.05 % ora_ckpt_TEST 12337 rt/4 root 0.00 K/s 2.62 K/s 0.00 % 0.00 % corosync -f and Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdaq 17.00 7.23 0.00 21 0 sdbf 17.33 7.44 0.00 22 0 sdcy 17.33 7.50 0.00 22 0 sddn 17.33 7.50 0.00 22 0 Thanks anyway ;-) Gianluca -------------- next part -------------- An HTML attachment was scrubbed... URL: From brilong at cisco.com Tue Jul 26 17:08:42 2016 From: brilong at cisco.com (Brian Long (brilong)) Date: Tue, 26 Jul 2016 17:08:42 +0000 Subject: [rhelv6-list] Limiting pvmove speed with control groups doesn't work as expected In-Reply-To: References: Message-ID: <98B3B294-B83D-444E-8C76-F7C7266A2414@cisco.com> Gianluca, This is an interesting thread and I?m glad you found a solution. I wonder if newer RHEL 6 kernels or if the RHEL 7 kernel provides the needed mechanism to tie pvmove to the kernel thread kcopyd such that your first technique would work without manually finding the kcopyd PID each time you call pvmove. /Brian/ -- Brian Long | | Research Triangle Park, NC . | | | . | | | . ' ' C I S C O On Jul 26, 2016, at 11:51 AM, Gianluca Cecchi > wrote: With iotop I see the PID of kcopyd that starts right after pvmove command.... (the kcopyd pid is different every pvmove run, not a fixed process) [root at dbatest1 ~]# ps -ef|grep 11139 root 11139 2 1 17:40 ? 00:00:02 [kcopyd] root 17103 19305 0 17:43 pts/0 00:00:00 grep 11139 BTW, kcopyd is child of kthreadd: root 2 0 0 Jun29 ? 00:00:00 [kthreadd] So I put kcopyd process into the control group too # echo 11139 > /cgroup/blkio/30M/tasks so that I now have pvmove process and kcopyd process [root at dbatest1 ~]# cat /cgroup/blkio/30M/tasks 11139 12765 -------------- next part -------------- An HTML attachment was scrubbed... URL: