<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Here’s what my replication agreements look like:</div><div class=""><br class=""></div><div class=""><div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list</div><div class=""><a href="http://sso-108.nym1.placeiq.net" class="">sso-108.nym1.placeiq.net</a>: master</div><div class=""><a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a>: master</div><div class=""><a href="http://sso-107.nym1.placeiq.net" class="">sso-107.nym1.placeiq.net</a>: master</div><div class=""><a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a>: master</div><div class=""><br class=""></div><div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list <a href="http://sso-107.nym1.placeiq.net" class="">sso-107.nym1.placeiq.net</a></div><div class=""><a href="http://sso-108.nym1.placeiq.net" class="">sso-108.nym1.placeiq.net</a>: replica</div><div class=""><a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a>: replica</div><div class=""><br class=""></div><div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list <a href="http://sso-108.nym1.placeiq.net" class="">sso-108.nym1.placeiq.net</a></div><div class=""><a href="http://sso-107.nym1.placeiq.net" class="">sso-107.nym1.placeiq.net</a>: replica</div><div class=""><a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a>: replica</div><div class=""><br class=""></div><div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a></div><div class=""><a href="http://sso-108.nym1.placeiq.net" class="">sso-108.nym1.placeiq.net</a>: replica</div><div class=""><a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a>: replica</div><div class=""><br class=""></div><div class="">[root@sso-107 (NY) ~]$ ipa-replica-manage list <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a></div><div class=""><a href="http://sso-107.nym1.placeiq.net" class="">sso-107.nym1.placeiq.net</a>: replica</div><div class=""><a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a>: replica</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">SSO-107</div><div class=""><br class=""></div><div class="">top - 15:58:08 up 2 days, 10:00, 1 user, load average: 0.00, 0.03, 0.06</div><div class="">Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie</div><div class="">Cpu(s): 12.2%us, 1.1%sy, 0.0%ni, 86.7%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st</div><div class="">Mem: 2952788k total, 2160216k used, 792572k free, 182584k buffers</div><div class="">Swap: 4094972k total, 0k used, 4094972k free, 678292k cached</div><div class=""><br class=""></div><div class=""> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div class="">11615 dirsrv 20 0 2063m 843m 19m S 25.5 29.3 403:53.56 ns-slapd</div><div class=""><br class=""></div><div class="">[root@sso-107 (NY) /var/log/dirsrv/slapd-PLACEIQ-NET]$ ls -al /proc/`cat /var/run/dirsrv/slapd-PLACEIQ-NET.pid`/fd|grep socket|wc -l</div><div class="">245</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">SSO-108</div><div class=""><br class=""></div><div class="">top - 15:57:26 up 3 days, 17:25, 1 user, load average: 0.03, 0.03, 0.00</div><div class="">Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie</div><div class="">Cpu(s): 0.3%us, 0.2%sy, 0.0%ni, 99.4%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st</div><div class="">Mem: 2952788k total, 2200792k used, 751996k free, 182084k buffers</div><div class="">Swap: 4094972k total, 0k used, 4094972k free, 713848k cached</div><div class=""><br class=""></div><div class=""> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div class="">24399 dirsrv 20 0 2055m 819m 19m S 0.8 28.4 54:48.53 ns-slapd</div><div class=""><br class=""></div><div class="">[root@sso-108 (NY) /var/log/dirsrv/slapd-PLACEIQ-NET]$ ls -al /proc/`cat /var/run/dirsrv/slapd-PLACEIQ-NET.pid`/fd|grep socket|wc -l</div><div class="">232</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">SSO-109</div><div class=""><br class=""></div><div class="">top - 16:00:05 up 4 days, 9:10, 1 user, load average: 0.06, 0.32, 0.35</div><div class="">Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie</div><div class="">Cpu(s): 0.7%us, 0.3%sy, 0.0%ni, 98.9%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st</div><div class="">Mem: 2952788k total, 2422572k used, 530216k free, 235472k buffers</div><div class="">Swap: 4094972k total, 0k used, 4094972k free, 906080k cached</div><div class=""><br class=""></div><div class=""> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div class="">22522 dirsrv 20 0 2065m 772m 19m S 1.2 26.8 308:13.07 ns-slapd</div><div class=""><br class=""></div><div class="">[root@sso-109 (NY) ~]$ ls -al /proc/`cat /var/run/dirsrv/slapd-PLACEIQ-NET.pid`/fd|grep socket|wc -l</div><div class="">219</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">SSO-110</div><div class=""><br class=""></div><div class="">top - 16:07:54 up 14 days, 18:03, 1 user, load average: 0.00, 0.01, 0.00</div><div class="">Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie</div><div class="">Cpu(s): 2.0%us, 1.0%sy, 0.0%ni, 96.7%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st</div><div class="">Mem: 2952788k total, 2304556k used, 648232k free, 155216k buffers</div><div class="">Swap: 4094972k total, 64k used, 4094908k free, 748972k cached</div><div class=""><br class=""></div><div class=""> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div class=""> 2401 dirsrv 20 0 2074m 839m 18m S 4.8 29.1 48:25.58 ns-slapd</div><div class=""> </div><div class="">[root@sso-110 (NY) /var/log/dirsrv/slapd-PLACEIQ-NET]$ ls -al /proc/`cat /var/run/dirsrv/slapd-PLACEIQ-NET.pid`/fd|grep socket|wc -l</div><div class="">257</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="Apple-interchange-newline"><br class=""></div><div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><table cellpadding="0" cellspacing="0" style="color: rgb(34, 34, 34); margin: 0px; font-size: 13.63636302948px; font-family: Times; background-color: rgb(255, 255, 255);" class=""><tbody class=""><tr height="3" class=""><td style="font-family: arial, sans-serif; margin: 0px; vertical-align: top;" class=""></td></tr></tbody><tbody class=""><tr height="3" class=""><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(68, 54, 124);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(99, 80, 178);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(148, 145, 145);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(203, 203, 203);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(148, 145, 145);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(99, 80, 178);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(68, 54, 124);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(99, 80, 178);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(148, 145, 145);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(203, 203, 203);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(148, 145, 145);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(99, 80, 178);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td width="5" style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td><td style="font-family: arial, sans-serif; margin: 0px; vertical-align: top; background-color: rgb(119, 117, 171);" class=""></td></tr></tbody><tbody class=""><tr height="10" class=""><td colspan="36" style="font-family: Georgia, serif; margin: 0px; vertical-align: top; padding-top: 5px; font-style: italic; font-size: 10px; letter-spacing: 0.1em; line-height: 1.5em; color: rgb(117, 107, 172);" class=""><span style="color: rgb(70, 70, 70);" class="">Jim Richard | </span><a href="http://www.google.com/url?q=http%3A%2F%2Fwww.placeiq.com%2F&sa=D&sntz=1&usg=AFrqEzcYjZpDPyqW7feNK9EgLq-c9JlHiw" target="_blank" style="color: rgb(117, 107, 172); text-decoration: none;" class="">PlaceIQ</a> | Systems Administrator | jrichard<a href="mailto:name@placeiq.com" target="_blank" style="color: rgb(117, 107, 172); text-decoration: none;" class="">@placeiq.com</a> | <a style="color: rgb(17, 85, 204);" class="">+1 (646) 338-8905</a></td></tr></tbody></table><div class=""><br class=""></div></div><br class="Apple-interchange-newline">
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Feb 19, 2015, at 9:33 AM, Rich Megginson <<a href="mailto:rmeggins@redhat.com" class="">rmeggins@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">On 02/18/2015 11:05 PM, Jatin Nansi
wrote:<br class="">
</div>
<blockquote cite="mid:54E57D28.103@redhat.com" type="cite" class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
Check the ns-slapd access and error logs of the DS instance
hosting the IPA instance. The strace output indicates that
ns-slapd spent most of its time waiting for network activity to
happen (poll), which is normal for ns-slapd.<br class="">
</blockquote>
<br class="">
The number of open connections correlates to the CPU usage. Do
this:<br class="">
<br class="">
# ls -al /proc/`cat /var/run/dirsrv/slapd-MY-DOMAIN.pid`/fd|grep
socket|wc -l<br class="">
<br class="">
How many socket connections do you have?<br class="">
<br class="">
Also, it will be very useful to get some stack traces of the running
server to see what the various threads are doing. See
<a class="moz-txt-link-freetext" href="http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs">http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs</a><br class="">
<br class="">
<blockquote cite="mid:54E57D28.103@redhat.com" type="cite" class=""> <br class="">
Jatin<br class="">
<div class="moz-cite-prefix">On 19/02/15 15:52, Jim Richard wrote:<br class="">
</div>
<blockquote cite="mid:30BE1A3D-7170-485D-9A91-C868682450D9@placeiq.com" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
I’ve got 4 Redhat IDM masters in a multi-master
config. 3.0.0-42.el6.centos is the IPA version, 389-ds-base
version 1.2.11.15-48.el6_6, Centos 6.6
<div class=""><br class="">
</div>
<div class="">Monitoring established connections on port 389 and
dsInOps over time shows a consistent/even level of activity
however 2 of the 4 IPA servers show ever increasing CPI usage
by ns-slapd. One ns-slapd process will start to show increased
CPU for a time, then drop off as another then increases. This
cycle continues with each switch seeing more and more total
SPU usage by ns-slapd.</div>
<div class=""><br class="">
</div>
<div class="">strace timing for the offending ns-slapd looks
like the following:</div>
<div class=""><br class="">
</div>
<div class=""><br class=""></div></blockquote></blockquote></div></div></blockquote><br class=""><blockquote type="cite" class=""><div class=""><div bgcolor="#FFFFFF" text="#000000" class=""><blockquote cite="mid:54E57D28.103@redhat.com" type="cite" class=""><blockquote cite="mid:30BE1A3D-7170-485D-9A91-C868682450D9@placeiq.com" type="cite" class=""><div class="">
</div>
<div class="">
<div class="">% time seconds usecs/call calls
errors syscall</div>
<div class="">------ ----------- ----------- ---------
--------- ----------------</div>
<div class=""> 96.12 9.342272 1133 8243
poll</div>
<div class=""> 3.86 0.375457 53 7066
41 futex</div>
<div class=""> 0.01 0.000668 0 8244
8244 getpeername</div>
<div class=""> 0.00 0.000374 0 929
close</div>
<div class=""> 0.00 0.000368 0 3201
read</div>
<div class=""> 0.00 0.000151 0 882
setsockopt</div>
<div class=""> 0.00 0.000095 2 42
access</div>
<div class=""> 0.00 0.000033 0 1365
fcntl</div>
<div class=""> 0.00 0.000000 0 42
open</div>
<div class=""> 0.00 0.000000 0 39
stat</div>
<div class=""> 0.00 0.000000 0 42
fstat</div>
<div class=""> 0.00 0.000000 0 1
madvise</div>
<div class=""> 0.00 0.000000 0 441
accept</div>
<div class=""> 0.00 0.000000 0 441
getsockname</div>
<div class=""> 0.00 0.000000 0 1
restart_syscall</div>
<div class="">------ ----------- ----------- ---------
--------- ----------------</div>
<div class="">100.00 9.719418 30979
8285 total</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I have carefully reviewed cn=config settings on
all four master servers to confirm that they match.</div>
<div class=""><br class="">
</div>
<div class="">Based on this strace output can you perhaps point
me in the right direction, give me a clue on what I should be
looking at.</div>
<div class=""><br class="">
</div>
<div class="">Here’s a screen shot of my Zabbix reporting to
help describe the problem. Note the graph in the bottom right
corner.</div>
<div class=""><br class="">
</div>
<div class="">The problem is most certainly related to
replication but I just don’t know what specifically to look
at.</div>
<div class=""><br class="">
</div>
<div class=""><span id="cid:part1.04040701.00090201@redhat.com"><Mail Attachment.png></span></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thanks in advance for any clues you can provide.<br class="">
<div class="">
<div style="font-family: Verdana; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="Apple-interchange-newline">
<br class="">
</div>
<div style="font-family: Verdana; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<table style="color: rgb(34, 34, 34); margin: 0px;
font-size: 13.63636302948px; font-family: Times;
background-color: rgb(255, 255, 255);" class="" cellpadding="0" cellspacing="0">
<tbody class="">
<tr class="" height="3">
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top;" class=""><br class="">
</td>
</tr>
</tbody><tbody class="">
<tr class="" height="3">
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(68, 54, 124);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(99, 80, 178);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(148, 145, 145);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(203, 203, 203);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(148, 145, 145);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(99, 80, 178);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(68, 54, 124);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(99, 80, 178);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(148, 145, 145);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(203, 203, 203);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(148, 145, 145);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(99, 80, 178);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class="" width="5"><br class="">
</td>
<td style="font-family: arial, sans-serif; margin:
0px; vertical-align: top; background-color:
rgb(119, 117, 171);" class=""><br class="">
</td>
</tr>
</tbody><tbody class="">
<tr class="" height="10">
<td colspan="36" style="font-family: Georgia, serif;
margin: 0px; vertical-align: top; padding-top:
5px; font-style: italic; font-size: 10px;
letter-spacing: 0.1em; line-height: 1.5em; color:
rgb(117, 107, 172);" class=""><span style="color:
rgb(70, 70, 70);" class="">Jim Richard | </span><a moz-do-not-send="true" href="http://www.google.com/url?q=http%3A%2F%2Fwww.placeiq.com%2F&sa=D&sntz=1&usg=AFrqEzcYjZpDPyqW7feNK9EgLq-c9JlHiw" target="_blank" style="color: rgb(117, 107,
172); text-decoration: none;" class="">PlaceIQ</a> |
Systems Administrator | jrichard<a moz-do-not-send="true" href="mailto:name@placeiq.com" target="_blank" style="color: rgb(117, 107, 172);
text-decoration: none;" class="">@placeiq.com</a> | <a moz-do-not-send="true" style="color: rgb(17, 85,
204);" class="">+1 (646) 338-8905</a></td>
</tr>
</tbody>
</table>
<div class=""><br class="">
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
</blockquote>
<br class="">
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
</blockquote>
<br class="">
</div>
-- <br class="">Manage your subscription for the Freeipa-users mailing list:<br class=""><a href="https://www.redhat.com/mailman/listinfo/freeipa-users" class="">https://www.redhat.com/mailman/listinfo/freeipa-users</a><br class="">Go To http://freeipa.org for more info on the project</div></blockquote></div><br class=""></div></div></div></body></html>