<div dir="ltr">Hello,<div style>I have a setup with postgresql 8.4 and 150 clients.</div><div style>My DB dump (using pg_dump, with -Fc option) is about 1.1 GB.</div><div style>I still haven't noticed slower operations.</div>
<div style>Should I planify VACUUM ANALYZE operations once a month? Could it lower the size of the dump?</div><div style>Would the script be like:</div><div style>spacewalk-service stop</div><div style>su --command "psql -c 'VACUUM VERBOSE ANALYZE;' -d spaceschema" postgres<br>
</div><div style>spacewalk-service start<br></div><div style>?</div><div style><br></div><div style>Thanks in advance for your help,</div><div style><br></div><div style>Pierre</div><div style><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">2013/4/11 Anton Pritchard-Meaker <span dir="ltr"><<a href="mailto:anton.pritchard-meaker@kit-digital.com" target="_blank">anton.pritchard-meaker@kit-digital.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-size:10pt;font-family:Tahoma">
<p>Thanks I really appreciate this, I'll definitely look into these actions. Downtime is not an issue for my Spacewalk implementation.</p>
<p> </p>
<p>I'm pretty new to PostreSQL, so I was completely unaware of all of maintenance tools available which actually sound quite necessary.</p>
<p> </p>
<div>
<p> </p>
<div><font><span style="FONT-SIZE:10pt">
<div>Anton Pritchard-Meaker | Unix Engineer<br>
<br>
</div>
</span></font></div>
</div>
<div style="font-size:16px;font-family:Times New Roman">
<hr>
<div style="DIRECTION:ltr"><font color="#000000" face="Tahoma"><b>From:</b> <a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a> [<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a>] on behalf of Paul Robert Marino [<a href="mailto:prmarino1@gmail.com" target="_blank">prmarino1@gmail.com</a>]<br>
<b>Sent:</b> 10 April 2013 22:39<div><div class="h5"><br>
<b>To:</b> <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a><br>
<b>Subject:</b> Re: [Spacewalk-list] API calls for new hosts<br>
</div></div></font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>In PostgreSQL 8.x auto vacuuming was first being introduced and the default settings weren't Ideal. Further more it wasn't a complete implementation so standard vacuuming is still necessary in 8.x<br>
<br>
</div>
In PostgreSQL 9.x auto vacuuming matured quite a bit a and manual vacuuming is needed far less often, but still a good idea to do occasionally.<br>
<br>
</div>
There are two kinds of vacuuming a lazy vacuum and a full vacuum.<br>
</div>
There are also two other table maintenance task which need to be done periodically as well.
<br>
</div>
<div><br>
A lazy vacuum does not require an exclusive table lock so in many cases may be executed while the database is actively in use; however there tend to be tables in spacewalk that constantly have lock which may hang the process so its best to schedule occasional
downtime for this operation. The good new is if you do it on a regular basis a lazy vacuum is quick. In addition in PostgreSQL 9.x the auto vacuum process fairly effectively opportunistically tries to do this for you as needed with as little impact as possible.<br>
<br>
</div>
A full vacuum requires an exclusive table lock but does a few things a lazy vacuum can't. The first thing it does is it flattens the MVCC ( MVCC is version control for rows it provides rollback capabilities and allows long running queries to complete without
the results being tainted by data added or deleted after the long running query was started). the MVCC needs to be occasionally flattened on high volume tables to prevent the version numbers from wrapping around (which can potentially cause a sort of data
corruption); however this is rare and may databases run for years without having to worry about this. The major advantage is that a Full vacuum can reclaim all of the disk space being used by old row versions. the lazy vacuum can only mark the space into a
pool for recycling (Oracle had the same thing literally called it the trash bin last time I worked with it) unless they are at the end of the last table file, also in PostgreSQL 8.x the developers realized the maximum size limit of recycle pool was too small
for modern databases so it was increased significantly in 9.x.<br>
<br>
<br>
</div>
<div>NOTE: a dump and load has the same effect as a full vacuum<br>
<br>
</div>
<div>ANALYZE<br>
<br>
</div>
<div>Analyzing updates your table statistics. the statistics are used by the query planner. what the query planner does is it takes the queries you run on the tables and re-optimizes them based on the table structure, the fragmentation level of the table, the
types of sorts, filters the query has, the indexes available and how efficient they, are more. the statistics tell the planer how efficient different types of operations are based on a series of test queries it executed the last time they were updated.<br>
</div>
<div>Analyzing is a non blocking operation however just like lazy vacuuming it can get hung up by other queries from spacewalk indefinitely, so its best to do it occasionally with spacewalk offline.<br>
</div>
<div>Analyzing can be done as part of a vacuum or independently. If done independently you can control it to the level where you can even tell it just to analyze a specific column; however its usually best to do an analyze with a vacuum for most people, only
very experienced DBAs should consider doing more advanced versions of the ANALYZE command .<br>
<br>
NOTE: a dump and load does not do an ANALYZE on the tables.<br>
<br>
<br>
</div>
<div>REINDEX<br>
<br>
</div>
<div>Vacuuming cleans up the table but not cleanup, defragment, or resort the indexes so it is important to at least once a year do a REINDEX on standard indexes to maintain performance, and more often for ordered indexes. A REINDEX can not be done as part
of a vacuum it is an independent operation. A REINDEX is an exclusive locking operation and as such can not be done at the same time as any thing else is accessing the table, as such spacewalk should be offline during this operation. reindexing is the slowest
maintenance operation and should only be done after a full vacuum. You should also do an ANALYZE after a REINDEX.<br>
<br>
</div>
<div>NOTE: a dump and load has the same effect as a REINDEX.<br>
<br>
<br>
</div>
<div>All of these operations are at the table level except the ANALYZE which may be done down to the column level. a REINDEX can also be done in the specific index level I think; however its usually most efficient to do the whole table at once unless you have
an unusually large table.<br>
<br>
</div>
<div>Finally there are command line tools for vaccum and reindex that can operate by sequentially cycling through the tables in the database; however if your disks ram and CPU can handle it you can run these operations in parallel on different tables to speed
things up via multiple SQL connections.<br>
<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>
<div><br>
<br>
<br>
<br>
<div>
<div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Apr 10, 2013 at 3:20 PM, Jon Miller <span dir="ltr">
<<a href="mailto:jonebird@gmail.com" target="_blank">jonebird@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">
<div dir="ltr">Perhaps routine Postgres maintenance was missing? I'm not proclaiming to be a Postgresql expert but do recall that periodic vacuuming[1] of the database is required / recommended? Your act of dropping and recreating sounds like side stepping
what could have been accomplished via maintenance.
<div><br>
</div>
<div>[1]: <a href="http://wiki.postgresql.org/wiki/VACUUM_FULL" target="_blank">http://wiki.postgresql.org/wiki/VACUUM_FULL</a><br>
<div><br>
</div>
<div>-- Jon Miller</div>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
<div>
<div>On Wed, Apr 10, 2013 at 8:45 AM, Anton Pritchard-Meaker <span dir="ltr">
<<a href="mailto:anton.pritchard-meaker@kit-digital.com" target="_blank">anton.pritchard-meaker@kit-digital.com</a>></span> wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">
<div>
<div>
<div lang="EN-GB">
<div>
<p class="MsoNormal"><span style="FONT-SIZE:11pt;FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d">I managed to fix this by exporting the database, dropping it in postgresql, re-creating it and then re-importing. Performance seems much better on the script
too. Call times have halved!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="FONT-SIZE:11pt;FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d"><u></u><u></u></span> </p>
<div>
<div style="BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;BORDER-BOTTOM:medium none;PADDING-BOTTOM:0cm;PADDING-TOP:3pt;PADDING-LEFT:0cm;BORDER-LEFT:medium none;PADDING-RIGHT:0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="FONT-SIZE:10pt;FONT-FAMILY:'Tahoma','sans-serif'">From:</span></b><span lang="EN-US" style="FONT-SIZE:10pt;FONT-FAMILY:'Tahoma','sans-serif'">
<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a> [mailto:<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a>]
<b>On Behalf Of </b>Anton Pritchard-Meaker<br>
<b>Sent:</b> 10 April 2013 15:17</span></p>
<div>
<div><br>
<b>To:</b> <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a><br>
<b>Subject:</b> Re: [Spacewalk-list] API calls for new hosts<u></u><u></u></div>
</div>
<p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal"><span style="FONT-SIZE:11pt;FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d">This script works nicely for existing hosts, the problem only occurs when I subscribe a new host and run the script/call. As an example, an existing host
with 36 updates available takes 4.7 seconds to get a result from Spacewalk whereas the new host below is taking 406.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="FONT-SIZE:11pt;FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d"><u></u><u></u></span> </p>
<p class="MsoNormal"><b><span lang="EN-US" style="FONT-SIZE:10pt;FONT-FAMILY:'Tahoma','sans-serif'">From:</span></b><span lang="EN-US" style="FONT-SIZE:10pt;FONT-FAMILY:'Tahoma','sans-serif'">
<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">spacewalk-list-bounces@redhat.com</a> [<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">mailto:spacewalk-list-bounces@redhat.com</a>]
<b>On Behalf Of </b>Paul Robert Marino<br>
<b>Sent:</b> 10 April 2013 15:03<br>
<b>To:</b> <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a><br>
<b>Subject:</b> Re: [Spacewalk-list] API calls for new hosts<u></u><u></u></span></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal" style="MARGIN-BOTTOM:12pt">It means your client timed out the connection.<br>
I assume this is Perl if so than you need to look at LWPs documentation to tune the time out parameter.<br>
<br>
<u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif';COLOR:#999999">-- Sent from my HP Pre3<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="FONT-FAMILY:'Verdana','sans-serif';COLOR:navy"><u></u><u></u></span> </p>
<div>
<div class="MsoNormal"><span style="FONT-FAMILY:'Verdana','sans-serif';COLOR:navy">
<hr align="left" size="3" width="75%">
</span></div>
</div>
<p class="MsoNormal" style="MARGIN-BOTTOM:12pt"><span style="FONT-FAMILY:'Verdana','sans-serif';COLOR:navy">On Apr 10, 2013 9:52 AM, Anton Pritchard-Meaker <<a href="mailto:anton.pritchard-meaker@kit-digital.com" target="_blank">anton.pritchard-meaker@kit-digital.com</a>>
wrote: </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="COLOR:#1f497d">Not too sure why, but a fresh request has generated the following log entries, but still no output and a 500 timeout returned:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"></span><u></u><u></u> </p>
<p class="MsoNormal"><span style="COLOR:#1f497d">Api logs:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"></span><u></u><u></u> </p>
<p class="MsoNormal"><span style="COLOR:#1f497d">[2013-04-10 14:20:02,951] INFO - REQUESTED FROM: 192.168.131.146 CALL: system.listLatestUpgradablePackages(4267x7b6a0781772f903417626f29664317c0, 1000010137) CALLER: (******) TIME: 406.117 seconds</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"></span><u></u><u></u> </p>
<p class="MsoNormal"><span style="COLOR:#1f497d">Tomcat:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"></span><u></u><u></u> </p>
<p class="MsoNormal"><span style="COLOR:#1f497d">10-Apr-2013 14:20:02 org.apache.jk.core.MsgContext action</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d">WARNING: Error sending end packet</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d">java.net.SocketException: Broken pipe</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at java.net.SocketOutputStream.socketWrite0(Native Method)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:538)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:127)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.jk.core.MsgContext.action(MsgContext.java:302)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.coyote.Response.action(Response.java:183)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.coyote.Response.finish(Response.java:305)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:205)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"> at java.lang.Thread.run(Thread.java:679)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d">10-Apr-2013 14:20:02 org.apache.jk.common.ChannelSocket processConnection</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d">WARNING: processCallbacks status 2</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"></span><u></u><u></u> </p>
<p class="MsoNormal"><span style="COLOR:#1f497d">Are there any other logs I could have a look at?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="COLOR:#1f497d"></span><u></u><u></u> </p>
<p class="MsoNormal"><span style="COLOR:#1f497d"></span><u></u><u></u> </p>
<div>
<div style="BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;BORDER-BOTTOM:medium none;PADDING-BOTTOM:0cm;PADDING-TOP:3pt;PADDING-LEFT:0cm;BORDER-LEFT:medium none;PADDING-RIGHT:0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="FONT-SIZE:10pt;FONT-FAMILY:'Tahoma','sans-serif'">From:</span></b><span lang="EN-US" style="FONT-SIZE:10pt;FONT-FAMILY:'Tahoma','sans-serif'"> Anton Pritchard-Meaker
<br>
<b>Sent:</b> 09 April 2013 16:42<br>
<b>To:</b> <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a><br>
<b>Subject:</b> API calls for new hosts</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">This one was happening in 1.8 and 1.9 on my RHEL5 based Spacewalk installation. I have a script to prepare update details for audit that works nicely with hosts already registered, but newly registered ones don’t work – returns a 500 timeout:<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">my $systems = $client->call('system.searchByName', $session, $search);
<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">returns the name ok:<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">[2013-04-09 16:23:52,714] INFO - REQUESTED FROM: 192.168.131.146 CALL: system.searchByName(4251x1ef1784c1d24abde4de1b183305f7458, *******) CALLER: (******) TIME: 0.351 seconds<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">my $kernel = $client->call('system.getRunningKernel', $session, $system->{'id'});<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">returns the kernel ok:<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">[2013-04-09 16:23:52,738] INFO - REQUESTED FROM: 192.168.131.146 CALL: system.getRunningKernel(4251x1ef1784c1d24abde4de1b183305f7458, 1000010137) CALLER: (******) TIME: 0.014 seconds<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">The next call is <u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">my $packages = $client->call('system.listLatestUpgradablePackages', $session, $system->{'id'});<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">At this point I get a “500 read timeout”.<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">All details are returning fine via the GUI. The script is here -
<a href="http://hastebin.com/tesovipoki.pl" target="_blank">http://hastebin.com/tesovipoki.pl</a>
<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">Any suggestions would be great – I don’t get anything in the tomcat logs unfortunately.<u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal">Cheers, <u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
<p class="MsoNormal" style="MARGIN-BOTTOM:10pt;LINE-HEIGHT:12.75pt"><b><span lang="EN-IN" style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif';COLOR:#365f91">Anton Pritchard-Meaker</span></b><span lang="EN-IN" style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif';COLOR:#4f81bd"> </span><span lang="EN-IN" style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif'">|
Unix Engineer</span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-IN" style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif';COLOR:#365f91">KIT digital</span></b><b><span lang="EN-IN" style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif';COLOR:#4f81bd"> </span></b><span lang="EN-IN" style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif'">|
York | </span><span><a href="http://www.kitd.com/" target="_blank"><span lang="EN-IN" style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif'">www.kitd.com</span></a></span><span lang="EN-IN" style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif'"> |
</span><span style="FONT-SIZE:9pt;FONT-FAMILY:'Arial','sans-serif'">The Future of Television</span><u></u><u></u></p>
<p class="MsoNormal"><u></u><u></u> </p>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
_______________________________________________<br>
Spacewalk-list mailing list<br>
<a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><br>
</blockquote>
</div>
<br>
</div>
<br>
_______________________________________________<br>
Spacewalk-list mailing list<br>
<a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
Spacewalk-list mailing list<br>
<a href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><br></blockquote></div><br></div>