<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none;"></style> </head> <body dir="ltr"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> I would like to say thank you to Jeremy, Paul, Robert and all who helped me with this issue. After much banging of the head on the wall, I finally got this working. <div> </div> <div>Thank you once again. I really appreciate the help, and again, I apology for any angry tones that I may have posted previously. My frustration got the better of me, and I don't want anyone on the list thinking that is my normal demeanor.</div> <div> </div> <div>Daryl</div> <div> </div> <div> </div> <div style="color: rgb(0, 0, 0);"> <hr tabindex="-1" style="display:inline-block; width:98%"> <div id="divRplyFwdMsg" dir="ltr">From: spacewalk-list-bounces@redhat.com <spacewalk-list-bounces@redhat.com> on behalf of jma@schaubroeck.be <jma@schaubroeck.be> Sent: Thursday, January 5, 2017 4:52 AM To: spacewalk-list@redhat.com Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from parentchannel <div> </div> </div> <div> Hi Daryl A few pointers: <ul> <li>CA Cert: should be under /etc/rhsm/ca/redhat-uep.pem as your system is registered with RHN. (if not, either use "find" or another tool to look for it, or use google, you should be able to download it) </li><li>SSL Client Cert and SSL Client Key: these are both in the entitlement certificates you can download. You can choose to split them in two files or just keep them together (both should work iirc) </li></ul> Add all 3 under Systems -> Kickstart -> GPG and SSL keys (choose SSL as type) and give them a clear name. Then you will be able to choose them in the dropdown menu for the repo config. Regards, Jeremy <div class="moz-cite-prefix">On 4/01/2017 16:16, Daryl Rose wrote: </div> <blockquote type="cite"> <div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif"> Hello Jeremy, Thank you for the information, and sorry for the late reply. Holidays and a heavy work load prevented me from getting back to this topic. Actually, you are the first person to specifically answer some of these questions. I do specifically remember asking the question, more than once, what am I missing? So, I believe the answer to that question is the SSL keys. I don't remember when I setup the original server setting up keys, but I must have if it had worked. I've read through the documentation, and I've read the subscription-manager man page, but I'm not really sure how to get the keys and what to do with them once I do get them. I am able to download the server entitlement from the Red Hat subscription, but I'm not sure if that is what I need or not. Also, I'm not familiar enough with certificates and I don't want to start trying commands that I am not familiar with and break my current configuration, which I know can happen. When I look at the repository page in the WUI, I see three SSL settings. <ul> <li>SSL CA Certificate </li><li>SSL Client Certificate </li><li>SSL Client Key </li></ul> <div>I'm assuming these are what I have to set, correct? In the drop down, there is only a single choice, </div> <div> <ul> <li>RHN-ORG-TRUSTED-SSL-CERT </li></ul> <div>I tried selecting that in the past, but that did not resolve this issue. Do I have to install the entitlement that I downloaded from Red Hat into the RHN-ORG-TRUSTED-SSL-CERT? </div> </div> <div> </div> <div>If someone can point me in the right direction of how to get the certs, and how to add them into Spacewalk without breaking my current configuration, I would appreciate it.</div> <div> </div> <div>Thank you.</div> <div> </div> <div>Daryl</div> <div> </div> <div style="color:rgb(0,0,0)"> <hr tabindex="-1" style="display:inline-block; width:98%"> <div id="divRplyFwdMsg" dir="ltr">From: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list-bounces@redhat.com"> spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> on behalf of <a class="moz-txt-link-abbreviated" href="mailto:jma@schaubroeck.be"> jma@schaubroeck.be</a> <a class="moz-txt-link-rfc2396E" href="mailto:jma@schaubroeck.be"> <jma@schaubroeck.be></a> Sent: Tuesday, December 20, 2016 10:29 AM To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from parentchannel <div> </div> </div> <div> Hi Daryl At the risk of repeating something someone else has already said, here's my thoughts: Spacewalk-repo-sync and the subscription manager registration are two separate things. I don't think anyone doubts that your server is registered all fine with RedHat and that all the relevant yum commands work. But as far as I know the two are unrelated. If you want to sync directly with Spacewalk itself (spacewalk-repo-sync is that, and it uses the repo config you set within spacewalk) then you will have to use the way Paul described: <blockquote type="cite"> 2) using a the same URL and keys as subscription manager, the URLs for the repos and the SSL keys can be retrieved from subscritpion manager on a registered host or through access.redhat.com by drilling into the subscriptions. the scheduling and exact options like the URL and if a SSL cert is used are configured on each repository through the spacewalk web interface. by the way this is the url you were being asked about. </blockquote> Aka: in your repository configuration in spacewalk you have to enter the correct URL, along with an SSL CA Certificate, SSL Client Certificate and SSL Client Key. The info where to get these and how to use them has been posted on the list a few times. Now the URL should be fine from what I read, but did you check the SSL cert/key? Are they the ones for your current registered server or for the old one? Are they still valid? (I had mine expire earlier this year, took me a while to figure out that was the issue) If there is an issue with these the sync will just outright fail without much useful info. Example: <blockquote type="cite">Sync started: Sun Nov 20 20:30:00 2016 ['/usr/bin/spacewalk-repo-sync', '--channel', 'rhel-6-server-x86_64', '--type', 'yum', ''] Repo URL: <whatever correct URL for whatever repo you want> ERROR: Cannot retrieve repository metadata (repomd.xml) for repository: rhel-6-server-x86_64. Please verify its path and try again Sync completed. Total time: 0:00:00</blockquote> Regards, Jeremy <div class="moz-cite-prefix">On 20/12/2016 16:44, Daryl Rose wrote: </div> <blockquote type="cite"> <div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif"> Robert, Thank you for the reply, and I apologize if I came off angry in my reply to you before. In your comments you kept asking about the URL. As I've stated many times previously, the server is registered, and entitled properly. I've been able to prove this many different ways. By running the yum command, by using the subscription-manager command to verify my entitlement etc.... I even opened a ticket with Red Hat to verify the entitlement, but because I mentioned Spacewalk in the ticket they told me to go pound sand because I don't have a Satellite subscription. But that was just a feel good ticket anyway. I know that the URL that I am using is accurate, because I can run the "yum repolist", "yum update" and "reposync" command and they all work. There is something else that is broke. You and others tell me to look at the logs, however, there is nothing in the logs to point me in the direction on what the issue is. I've posted the only error that I receive. Oh, one thing that I have failed to mention is that I disabled the schedule a couple of weeks ago when this issue first started. So that takes taskomatic out of the picture. I have been using the "spacewalk-repo-sync" command from the command line trying to troubleshoot this issue. That is why in all of my previous updates I kept specifying spacewalk-repo-sync. But that is my fault for not mentioning that before. With that said, I don't care how many verbose v's I put at the end of the command, the only error that I get is the error that I've been posting. Here is what I get in the channel log: 2016/12/20 09:12:22 -05:00 Sync of channel started. <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.channel_info('rhel-6-server-rpms',)</div> <div>2016/12/20 09:12:22 -05:00 Repo URL: <RHEL URL HERE></div> <div>2016/12/20 09:12:22 -05:00 Sync of channel completed in 0:00:00.</div> <div> </div> Here is what I get in the reposync.log: <div> <div>2016/12/20 09:12:22 -05:00 Command: ['/usr/bin/spacewalk-repo-sync', '--channel', 'rhel-6-server-rpms', '-vvv']</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(136,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(136, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(133,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(133, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(103,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(103, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(122,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(122, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(129,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(129, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(120,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(120, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(121,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(121, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(131,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(131, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(124,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(124, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(138,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(138, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(134,)</div> <div>2016/12/20 09:12:22 -05:00 2595 0.0.0.0: server/rhnChannel.isCustomChannel(134, 'is a custom channel')</div> <div>2016/12/20 09:12:22 -05:00 ======================================</div> <div>2016/12/20 09:12:22 -05:00 | Channel: rhel-6-server-rpms</div> <div>2016/12/20 09:12:22 -05:00 ======================================</div> <div>2016/12/20 09:12:22 -05:00 Sync of channel started.</div> <div> Please check 'reposync/rhel-6-server-rpms.log' for sync log of this channel.</div> <div>2016/12/20 09:12:22 -05:00 Sync of channel completed.</div> <div>2016/12/20 09:12:22 -05:00 Total time: 0:00:00</div> </div> <div>Not much information to go on. </div> <div> </div> <div>Thanks</div> <div> </div> <div>Daryl <hr tabindex="-1" style="font-size:12pt; display:inline-block; width:98%"> </div> <div> <div style="color:rgb(0,0,0)"> <div id="divRplyFwdMsg" dir="ltr">From: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list-bounces@redhat.com"> spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> on behalf of Robert Paschedag <a class="moz-txt-link-rfc2396E" href="mailto:robert.paschedag@web.de"> <robert.paschedag@web.de></a> Sent: Tuesday, December 20, 2016 9:02 AM To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> Cc: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from parentchannel <div> </div> </div> <div> <div style="font-family:Verdana; font-size:12.0px"> <div> <div> <div>Hi Daryl,</div> <div> </div> <div>I'm not angry....also frustrated because we all seem to talk about different things and you seem to focus on an problem, that - in my opinion - is not of interest yet.</div> <div> </div> <div>The first is to get the synchronisation from RHEL done!</div> <div> </div> <div>Now....in your first post, you wrote, that you did not get updates from RHEL since several month. This makes me think....aha....you are downloading "directly" from RHEL. Hence, this is your "master" spacewalk server. So the "creation" of the metadata of the "downloaded" packages (that does not work right now) is currently useless.</div> <div> </div> <div>The main problem is....why can't you download the packages?</div> <div> </div> <div>One problem might be...that you are "not" entitlet to download. But you wrote, you are. So I take this for granted.</div> <div> </div> <div>So....when I look into one of my sync logs, it looks like this...(these are stored in /var/log/rhn/reposync/...)</div> <div> </div> <div>...</div> <div>Sync started: Tue Mar 22 11:43:43 2016 ['/usr/bin/spacewalk-repo-sync', '--channel', 'sles12-sp1-updates-x86_64', '--type', 'yum'] Repo URL: <a href="http://s0002120.swr.ard/repo/SUSE/Updates/SLE-SERVER/12-SP1/x86_64/update/"> http://<MY-INTERNAL-SMT-Server>/repo/SUSE/Updates/SLE-SERVER/12-SP1/x86_64/update/</a> Packages in repo: 893 Packages already synced: 0 Packages to sync: 689 1/689 : openssh-fips-6.6p1-33.1-0.x86_64 2/689 : libxml2-tools-2.9.1-17.1-0.x86_64 3/689 : ceph-common-0.80.11-8.1-0.x86_64 4/689 : desktop-data-SLE-12-3.1-0.noarch 5/689 : postgresql94-docs-9.4.6-7.2-0.noarch 6/689 : gnome-shell-3.10.4-54.1-0.x86_64 7/689 : typelib-1_0-Polkit-1_0-0.113-5.6.1-0.x86_64 8/689 : xorg-x11-server-extra-7.6_1.15.2-46.1</div> <div>...</div> <div> </div> <div>So here the "Repo URL" the script tries to connect to, is listed. And this is the URL we want you to show us. Please also remove any possible authentication tokens and credentials.</div> <div> </div> <div>What I also think is that you - maybe - run a spacewalk proxy! that tries to download his packages from an "upstream" spacewalk server. But you did not mention this (or maybe a didn't read it).</div> <div> </div> <div>Regards,</div> <div>Robert</div> <div> </div> <div> </div> <div name="quote" style="margin:10px 5px 5px 10px; padding:10px 0px 10px 10px; border-left-color:rgb(195,217,229); border-left-width:2px; border-left-style:solid"> <div style="margin:0px 0px 10px">Gesendet: Dienstag, 20. Dezember 2016 um 15:16 Uhr Von: "Daryl Rose" <a class="moz-txt-link-rfc2396E" href="mailto:darylrose@outlook.com"> <darylrose@outlook.com></a> An: <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list@redhat.com">"spacewalk-list@redhat.com"</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list@redhat.com"><spacewalk-list@redhat.com></a> Betreff: Re: [Spacewalk-list] RHEL 6.x repository not syncing from parentchannel</div> <div name="quoted-content"> <div> <div id="divtagdefaultwrapper" style="color:rgb(0,0,0); font-family:Calibri ,Arial ,Helvetica ,sans-serif; font-size:12pt"> Hello Paul, Thank you for the information. In response to your statement about angry reply. No, I am not angry. Frustrated yes, angry no. Comments made by Robert, and then you do frustrate me. I've been trying to figure this out now for several weeks, posting questions with no answer. In more than one reply I had specifically said where I had obtained the URL. I've tried to be as specific as I could about what I am seeing and what specific steps I've taken to resolve this issue. I've asked multiple times, where does the data in the reposync directory come from? What creates it? What frustrates me is the few replies that I had received were not very clear as to where to look, or how to troubleshoot this product. I personally felt that Roberts response to me was laced with angry tones. In my reply to him, I tried to make it clear that the URL is not the issue, the missing data in the reposync directory is the issue. The error that I am receiving is miss leading. spacewalk-repo-sync can't find repond.xml because its not there, not because of the path being wrong, but because it and the other keys required are not there. The question that I've been asking, is WHAT CREATES THAT DATA? Also, one thing that you are correct about is that I don't have a lot of knowledge with the product. I have over 1500 servers that I have to maintain and keep running. I don't have a lot of time to dig into the product and learn every single little feature and component. That is why I post questions to forums, mailing lists and yes, Google. But, as of right now, I haven't been able to find anything that would help me resolve this issue. In response to the steps that you outlined below. I'll review them and see if I can use them to troubleshoot the product more. <snip> 2) using a the same URL and keys as subscription manager, the URLs for the repos and the SSL keys can be retrieved from subscritpion manager on a registered host or through access.redhat.com by drilling into the subscriptions. either way you debug it by looking at the output of the reposync command. normally reposync is executed by taskomatic and a log file is created with the output for each repository. in addition taskomatic has the output in its logs. the scheduling and exact options like the URL and if a SSL cert is used are configured on each repository through the spacewalk web interface. by the way this is the url you were being asked about. </snip> The URL that I am using IS the same URL that subscription manager uses. Again, I posted that several time before. As a matter of fact, in my troubleshooting, I ran the native reposync command and successfully downloaded the RHEL repo. So, this is why I kept focusing my attention on spacewalk-repo-sync. From looking at the spacewalk-repo-sync script, it appears that it calls the native reposync command, but I am not a python developer, and I can only follow the script so much. But I can see where it imports reposync, and then executes in within the script, but I can't debug the script, or trace the script, because I don't know python, nor do I know anyone who does that can help me figure out how it works. I'll look at some of the other items that you mentioned, but I really doubt that its a corrupt RPM, or a stuck taskomatic job. I've ran the spacewalk-data-fsck command more that once, and I've stopped started Spacewalk more than once and this problem is still an issue. I would still like to have one question answered; what creates the data in the /var/cache/rhn/reposync directory? e.g. repomd.xml, mirrorlist.txt, cachecookie and keys? I can copy the data over from the old server to the new server and work around this issue, but that is just a work around. It does not resolve the issue and I need to resolve the issue. So, if you or someone else on this list can answer that question, I think that I can use that to figure out how to resolve this. Something creates the data, I would like to know what. Thank you very much for your reply Paul, and thank you for the information. Daryl <div style="color:rgb(0,0,0)"> <div> <hr style="width:98%; display:inline-block"> <div id="x_divRplyFwdMsg">From: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list-bounces@redhat.com"> spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> on behalf of Paul Robert Marino <a class="moz-txt-link-rfc2396E" href="mailto:prmarino1@gmail.com"> <prmarino1@gmail.com></a> Sent: Monday, December 19, 2016 3:02 PM To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from parentchannel <div> </div> </div> </div> <div class="PlainText">Daryl Im going to lay out a plan for you to debug this situation, but since you gave angry responses when you obviously did not understand the questions you were being asked I think you really need to read the manuals. so I will point you in the right direction in a way so you will better get to know spacewalk. potential problem 1 - repos not syncing from Red Hat. there are 2 methods used to sync repos from RedHat. 1) Mrepo - this is the legacy method. mrepo is a python script which uses a username and password to mirror Red Hats repos locally, then spacewalk syncs from it using reposync. optionally an other script is also used to sync erratas with this because mrepo does not sync them. 2) using a the same URL and keys as subscription manager, the URLs for the repos and the SSL keys can be retrieved from subscritpion manager on a registered host or through access.redhat.com by drilling into the subscriptions. either way you debug it by looking at the output of the reposync command. normally reposync is executed by taskomatic and a log file is created with the output for each repository. in addition taskomatic has the output in its logs. the scheduling and exact options like the URL and if a SSL cert is used are configured on each repository through the spacewalk web interface. by the way this is the url you were being asked about. Potential problem 2 - repos are syncing but taskomatic doesn't update the meta data correctly. the primary cause for this is the repo-metadata job exceeding the memory limits configured in taskomatic. This is common known tuning issue with the RHEL 5 and 6 repositories because they are huge, and some other large repositories as well. the root cause is poor error handling in taskomatic's upstream code. if one job is killed in the middle then none of the future jobs will run until taskomatic is restarted. the first step to diagnosing it is to go into spacewalks admin tab and drill down into it to find the internally scheduled tasks which do not appear in the main scheduling tab. drill down into the history of the metadata job to see if it shows a job hung for a day or more (if so that job actually died). if there is one hung restart taskomatic and you should see it try to execute the tasks it missed. next you want to look at taskomatics logs to find out if any jobs exceded the maximum memory limits. if any have then you need to increase the limit in taskomitics configuration. other 2 potential problems which are less likely. 1) corrupt RPM files or database checksums, both can be diagnosed and fixed by the spacewalk-data-fsck command. 2) in some rare cases 2 RPM's with the same name and version but different checksums have been know to be put in the same channel the results in Yum and the Red hat installer saying the file cant be found or has a corrupt check sum. usually this was caused by a errats sync script which is not aware of a known bug in the errata API's which has never been fixed. the way to fix this to delete all packages with the offending name and version from the repository then re run reposync manually via the command line, or spacewalk web interface on the affected repos. for further reading on this you may want to look at spacewalks documentation and the Red hat documentation for RHN Satellite 5.x on access.redhat.com additionally all of these issues have been discussed many times on this mailing list and can be found easilly by a google search On Mon, Dec 19, 2016 at 2:54 PM, Daryl Rose <a class="moz-txt-link-rfc2396E" href="mailto:darylrose@outlook.com"> <darylrose@outlook.com></a> wrote: > Robert, > > > Perhaps I don't understand what you're telling me. > > > How do I fix problem 1? What steps do I have to do to fix problems 1? > > > I know that it's not syncing the remote repository, but the error that I am > seeing points to problem 2. To me, problem 2 is what is preventing problem > 1 from being achieved that is why I am focusing on problem 2. > > > Thanks > > > Daryl > > > ________________________________ > From: Robert Paschedag <a class="moz-txt-link-rfc2396E" href="mailto:robert.paschedag@web.de"> <robert.paschedag@web.de></a> > Sent: Monday, December 19, 2016 1:43 PM > > To: Daryl Rose; <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: AW: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > We are talking about 2 problems. > > 1. Your server doesn't sync the remote repository. > 2. Your server is not creating repository meta data. > > You have to fix problem 1 first. If that works, you can think about fixing > problem 2. > > From what you are telling.... You're trying to fix problem 2. I'm taking > about problem 1 all the time. > > Again... First fix number 1. > > Regards > Robert > > > ------ Originalnachricht------ > Von: Daryl Rose > Datum: Mo., 19. Dez. 2016 20:14 > An: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a>; > Cc: > Betreff:Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > Sorry about the reply. Normally when I reply, it does go to the group, not > just to the individual who sent the reply, but for whatever reason, I have > to specify the mailing list when replying to your emails. > > > The path that the error is referring to is: > > > /var/cache/rhn/reposync/content_dist_rhel_server_6_$releasever_$basearch_os/repomd.xml > > or > > /var/cache/rhn/reposync/content_dist_rhel_server_6_6Server_x86_64_os_/repomd.xml > > As I stated in the past, repomd.xml and additional content is not created > when spacewalk-repo-sync runs. I've been able to prove that > spacewalk-repo-sync creates this directory because I've removed it multiple > times and the directory is created and a sub-directory called "packages", > but that is it. None of the keys are created, and repomd.xml is not > created. > > spacewalk-repo-sync is a python script. I don't know anything about python > and I don't know how to trace the script. I've tried "phthon -m reposync > --trace spacewalk-repo-sync", but that doesn't work. I would really like to > see what is happening when the script tries to execute the reposync module, > but I get "reposync module doesn't exists". > > Maybe if I can see what is going on in the spacewalk-repo-sync script, then > I can figure out what is causing my issue. > > Thanks > > Daryl > ________________________________ > From: Robert Paschedag > > Sent: Monday, December 19, 2016 11:35 AM > To: Daryl Rose > Cc: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: AW: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > Please, answer to "all" so the list gets informed, too. > > And there is no path mentioned anywhere! Is there no path in the logs that > the script tries to connect to? > LikeHTTP://ftp.redhat.com/some/path/repomd.xml > > I don't care the name of the repository or path on your local machine.... > The download URL is the thing that matters... > > Regards > Robert > > ------ Originalnachricht------ > Von: Daryl Rose > Datum: Mo., 19. Dez. 2016 18:11 > An: Robert Paschedag; > Cc: > Betreff:Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > I tried both with and without variables. This is an example of without the > variables. Same error: > > > 11:08:18 ERROR: Cannot retrieve repository metadata (repomd.xml) for > repository: content_dist_rhel_server_6_6Server_x86_64_os_. Please verify its > path and try again > > > [root reposync]# ls -l > drwxr-xr-x. 3 root root 4096 Dec 19 11:08 > content_dist_rhel_server_6_6Server_x86_64_os_ > > Daryl > > ________________________________ > From: Robert Paschedag> > Sent: Monday, December 19, 2016 10:08 AM > To: Daryl Rose; <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: AW: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > Daryl, > > I don't think, that you can use the URL that is used by "yum" within > spacewalk. Just because of these variables in it! I think this is why you > get the "invalid path" errors. So just print also the paths within your logs > just before the error occurs to verify. Then... Try to use the "resolved" > URL within spacewalk. > > Regards > Robert > > > ------ Originalnachricht------ > Von: Daryl Rose > Datum: Mo., 19. Dez. 2016 16:20 > An: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a>; > Cc: > Betreff:Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > Robert, > > > The URL came from the /etc/yum.repos.d/redhat.repo.redhat.repo, which is > created when registering to Red Hat. The path in question is created when > I run the spacewalk-repo-sync command. I've removed that particular path > several times, modified the URL and every time I run the spacewalk-repo-sync > command, the directory in question get recreated. > > > I do know that there is a difference between the original reposync directory > and the new reposync directory. > > > This is the content of the reposync directory on the original server: > > > [root]# cd /var/cache/rhn/reposync > > [root reposync]# ls -l > total 104 > drwxr-xr-x. 3 root root 4096 Oct 12 11:18 centos-7-extra > drwxr-xr-x. 3 root root 4096 Nov 8 21:00 centos-7-updae > drwxr-xr-x. 3 root root 4096 Aug 24 12:10 centos_7_channel > drwxr-xr-x. 3 root root 4096 Aug 29 13:10 centos_7_plus > drwxr-xr-x. 3 root root 4096 Dec 9 14:03 rhel-6-server-rpms > drwxr-xr-x. 3 root root 4096 Aug 23 18:00 sles11-sp3-pool > drwxr-xr-x. 3 root root 4096 Aug 23 19:00 sles11-sp3-pool-x86 > drwxr-xr-x. 3 root root 4096 Nov 8 18:01 sles11-sp3-update > drwxr-xr-x. 3 root root 4096 Dec 8 19:00 sles11-sp3-update-pool-x86 > drwxr-xr-x. 3 root root 4096 Aug 23 20:00 sles11-sp4-pool > drwxr-xr-x. 3 root root 4096 Nov 8 20:00 sles11-sp4-update > > > This is the contents of the reposync directory on the new server: > > > [root]# cd /var/cache/rhn/reposync > > [root reposync]# ls -l > total 71328 > drwxr-xr-x. 3 root root 4096 Dec 18 17:00 centos_7_centosplus_x86_64_ > drwxr-xr-x. 3 root root 4096 Dec 18 18:00 centos_7_extras_x86_64_ > drwxr-xr-x. 3 root root 4096 Dec 18 17:30 centos_7_os_x86_64_ > drwxr-xr-x. 3 root root 4096 Dec 18 18:30 centos_7_updates_x86_64_ > drwxr-xr-x. 3 root root 4096 Dec 15 15:38 > content_dist_rhel_server_6_$releasever_$basearch_os > drwxr-xr-x. 3 root root 4096 Dec 18 20:30 > repo_$RCE_SLES11-SP3-Pool_sle-11-i586 > drwxr-xr-x. 3 root root 4096 Dec 18 19:30 > repo_$RCE_SLES11-SP3-Pool_sle-11-x86_64 > drwxr-xr-x. 3 root root 4096 Dec 18 21:00 > repo_$RCE_SLES11-SP3-Updates_sle-11-i586 > drwxr-xr-x. 3 root root 4096 Dec 18 20:00 > repo_$RCE_SLES11-SP3-Updates_sle-11-x86_64 > drwxr-xr-x. 3 root root 4096 Dec 18 21:30 > repo_$RCE_SLES11-SP4-Pool_sle-11-x86_64 > drwxr-xr-x. 3 root root 4096 Dec 18 22:00 > repo_$RCE_SLES11-SP4-Updates_sle-11-x86_64 > > Notice the difference in the sub-directories? The original server was SW > v2.5, the new server is SW v2.6. I'm assuming that the spacewalk-repo-sync > command changed the way the sub-directories were created. However these are > created shouldn't make any difference, because they all work, except the Red > Hat repo. > > > The path is correct. The issue is that repomd.xml and additional content is > not created when the sub-directory > "content_dist_rhel_server_6_$releasever_$basearch_os" is created, and I have > no idea how the contents is created. > > > The error message saying to check the path is not correct. The path is > correct, its just the contents is not there. > > > Daryl > > > ________________________________ > From: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list-bounces@redhat.com"> spacewalk-list-bounces@redhat.com</a>> on behalf of Robert Paschedag> > Sent: Friday, December 16, 2016 9:37 AM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > It says... The path is wrong. So what did you put into the URL line????? > > > > Am 16.12.2016 15:50 schrieb Daryl Rose>: > > So, today I decided to create a new RHEL 6 channel and repo and then run the > "spacewalk-repo-sync" command and see if I can't get this repo to populate > with packages from Red Hat. The same erroroccurred. > > > ERROR: Cannot retrieve repository metadata (repomd.xml) for repository: > content_dist_rhel_server_6_6Server_x86_64_os_. Please verify its path and > try again > > > This tells me that there isn't anything corrupt. I'm really think that I > missed something in the configuration or setup of the new server. > > > When I setup the original server, v2.3, I'm thinking there were > someadditional steps that I had to do once I registered the server to Red > Hat. I've been reviewing the installation instructions, but I don't see > anything extra that I missed. In the SW, v2.6, installation instructions, > all I see is to make sure that the server is properly registered to > theappropriate channel: > > > > Red Hat Optional Server (Red Hat Enterprise Linux) > > When using Red Hat Enterprise Linux 6 or 7, make sure you are subscribed to > the appropriate Red Hat Optional Server channel: > > Red Hat Optional Server 6 , OR > Red Hat Optional Server 7 > > > The server is properly registered and entitled. I can run a yum repolist > and see the repository. I can issue the subscription-manager command and > see that the server is subscribed and I can see what repos are enabled. But > for some reason, that is not getting passed onto the Spacewalk repo. > > > I tried opening a ticket with Red Hat just so I could verify that everything > was registered correctly, but since I don't have a satellite subscription, > they closed the ticket without even answering my question. > > > I've run out of idea's. If someone can give me more suggestions, I'm > willing to listen. > > > Thanks > > > Daryl > > > > ________________________________ > From: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list-bounces@redhat.com"> spacewalk-list-bounces@redhat.com</a>> on behalf of Robert Paschedag> > Sent: Friday, December 16, 2016 12:44 AM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > Oh... Just saw that the checksum error occurs when run on a "client".. > > > > Am 15.12.2016 22:48 schrieb Daryl Rose>: > > One more thought.... > > > What would happen if I were to delete the current RHEL channel and > repository and recreate them? What will happen to all of the packages and > clients currently registered to the channel/repo? I know that the packages > will become orphan, but can I put them back into the new repo, or do I just > have to remove them and re-download them from Red Hat? How about the > registered clients? Do I have to re-register them, or if I keep the same > activation key, the same channel/repo names, will they just pickup the > change? > > > Thanks > > > Daryl > > > > ________________________________ > From: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list-bounces@redhat.com"> spacewalk-list-bounces@redhat.com</a>> on behalf of Daryl Rose > > Sent: Thursday, December 15, 2016 2:59 PM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > > Hello Chris, > > > Thank you for the following steps. I think this resolved this immanent > issue, but the issue of not being able to sync from the parent Red Hat > repository is still an issue. > > > Do you know how the contents of ./cache/rhn/reposync are created? As a work > around, I copied the contents ofreposyncfrom the original server to the new > server, and I'm able to sync from the Red Hat repository, but that is not a > long term solution. > > > I thought that perhaps I had found a solution. Yum can access the > repository just fine so I copied the contents > from/var/cache/yum/x86_64/6Server/rhel-6-server-rpms to the reposync > directory. It started off just fine, but eventually failing due to not > being able to find some keys. Actually the keys were generated, but they > were of zero length and that won't work. > > > It seems to be obvious that there is something else wrong with my > installation, I just don't know what it is. If you, or anyone else have any > thoughts are suggestion, I'm willing to take a look and give it a try. > > > Thank you. > > > Daryl > > > ________________________________ > From: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list-bounces@redhat.com"> spacewalk-list-bounces@redhat.com</a>> on behalf of Snyder, Chris > > Sent: Wednesday, December 14, 2016 7:07 PM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > > The “metadata ….” Error indicates that your yum cache (provided by the > Spacewalk server) is out of with what is actually available through the > Spacewalk package database. This can be because taskomatic isn’t running or > there’s a problem somewhere. Check your logs under /var/log/rhn/ for any > issues with taskomatic or any other components. > > > > However, I’ve run into this before and I do the following to get around it > (NOTE: your paths will probably be different than mine): > > > > # Blow away all cached metadata > > rm -fr /data/pkg/cache/rhn/repodata/ > > > > # restart spacewalk so empty repodata cache directory is recreated. > > spacewalk-service stop > > spacewalk-service start > > > > # If not already, cache credentials forspacecmd (This assumes you have > configured ~/.spacecmd/config previously) > > spacecmd softwarechannel_list > > > > # Regenerate repo data forall channels > > for i in `spacecmd softwarechannel_list`; do spacecmd > softwarechannel_regenerateyumcache $i; done > > > > # Waited a few minutes fordata to regenerate > > > > Once that’s done and the metadata caches are updated (you’ll see updated > directories under the cache dir and you should see info in the taskomatic > logs), go to each registered spacewalk client and run > > > > yum clean all > > > > If this doesn’t work, I would have to say that you’ve got something else > wrong somewhere else. > > > > Good luck. > > > > From: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list-bounces@redhat.com"> spacewalk-list-bounces@redhat.com</a> > [<a href="mailto:spacewalk-list-bounces@redhat.com" target="_blank">mailto:spacewalk-list-bounces@redhat.com</a>]On Behalf Of Daryl Rose > Sent: Wednesday, December 14, 2016 5:58 PM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > > > Hello everyone, > > > > I have not received any replies from my previous posting, so I thought that > I would post an update hoping someone would see it and reply. > > > > This error seems to go deeper than just syncing from Red Hat. Server that > are currently registered to the Spacewalk server are not able to access the > RHEL repository on SW. If issuing the "yum update" command, we receive the > error: > > > > Error: failed to retrieve repodata/primary.xml.gz from rhel-6-server-rpms > > error was [Errno -1] Metadata file does not match checksum > > > > If I issue the "yum repolist" command, I only see "0" for packages. > > > > So, this issue is larger that what I initially thought. Anyone have any > thoughts or idea's? > > > > Daryl > > ________________________________ > > <a class="moz-txt-link-abbreviated" href="mailto:From:spacewalk-list-bounces@redhat.com"> From:spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> > on behalf of Daryl Rose <a class="moz-txt-link-rfc2396E" href="mailto:darylrose@outlook.com"> <darylrose@outlook.com></a> > Sent: Friday, December 9, 2016 2:56 PM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > > > So, I resurrected the old server from the grave and copied the contents of > "/var/cach/rhn/reposync/rhel-6-server-rpms" to > "/var/cach/rhn/reposync/content_dist_rhel_server_6_$releasever_$basearch_os" > and ran a sync. The sync worked. > > > > The sync worked, but it caused a new error. A team member registered a Red > Hat server just after I made these changes. The server registered, but > with the following error: > > > > Bad id for repo: logstash 5.x, byte = 8 > > > > I'm guessing that something within the data that I copied over does not > align with the repository on SW? > > > > Another thing that I noticed is that the directory structure in the old > reposync directory is different than the new server. For example: > > > > OLD: > > /var/cache/rhn/reposync/centos-7-extra > > /centos-7-server-rpms > > /centos-7-updae > > /centos_7_channel > > /centos_7_plus > > > > NEW: > > /var/cache/rhn/reposync/centos_7_centosplus_x86_64_ > > /centos_7_extras_x86_64_ > > /centos_7_os_x86_64_ > > /centos_7_updates_x86_64_ > > > > Is that a SW v2.3 vs. v2.6 change? > > > > Thanks > > > > Daryl > > > > ________________________________ > > <a class="moz-txt-link-abbreviated" href="mailto:From:spacewalk-list-bounces@redhat.com"> From:spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> > on behalf of Daryl Rose <a class="moz-txt-link-rfc2396E" href="mailto:darylrose@outlook.com"> <darylrose@outlook.com></a> > Sent: Thursday, December 8, 2016 10:32 AM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > > > More questions: > > > > When I run "spacewalk-repo-sync --channel=rhel-6-server-rpms", I get the > error: > > > > ERROR: Cannot retrieve repository metadata (repomd.xml) for repository: > content_dist_rhel_server_6_$releasever_$basearch_os. Please verify its path > and try again > > > What creates the content in > "/var/cache/rhn/reposync/content_dist_rhel_server_6_$releasever_$basearch_os"? > > > > All of the other channels that I created all have "cachecookie, packages, > primary.xml.gz, primary.xml.gz.sqlite, repomd.xml and updateinfo.xml.gz". > > > > There must be a process that creates this content, but what, I'm not sure. > How can I manually create this content? > > > > Thanks > > > > Daryl > > > > ________________________________ > > <a class="moz-txt-link-abbreviated" href="mailto:From:spacewalk-list-bounces@redhat.com"> From:spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> > on behalf of Daryl Rose <a class="moz-txt-link-rfc2396E" href="mailto:darylrose@outlook.com"> <darylrose@outlook.com></a> > Sent: Thursday, December 8, 2016 9:26 AM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > > > Quick update. > > > > Poking around on Red Hat I found a doc that suggested that I look at the > name in /etc/sysconfig/network and ensure that the hostname is FQDN and then > run "spacewalk-hostname-rename". Host name was not FQDN, so I renamed it, > but because I use a signed cert, will running spacewalk-hostname-rename > cause a problem with that cert? > > > > Thanks > > > > Daryl > > > > ________________________________ > > <a class="moz-txt-link-abbreviated" href="mailto:From:spacewalk-list-bounces@redhat.com"> From:spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> > on behalf of Daryl Rose <a class="moz-txt-link-rfc2396E" href="mailto:darylrose@outlook.com"> <darylrose@outlook.com></a> > Sent: Thursday, December 8, 2016 8:55 AM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > > > Robert, > > > > Yes, the server is registered and has an active subscription with Red Hat. > As a matter of fact, I just now re-registered to make sure that all is good. > I can see from the command line subscription manager that the server is > subscribed, and I confirmed that it is entitled. > > > > I use a "signed" cert so I'm wondering if the signed cert has something to > do with this? However, I found a troubleshooting document on Red Hat that > verifies that I am entitled and using the certificate provided by Red Hat to > communicate back to them. > > > > Thank you > > > > Daryl > > ________________________________ > > <a class="moz-txt-link-abbreviated" href="mailto:From:spacewalk-list-bounces@redhat.com"> From:spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> > on behalf of Robert Paschedag <a class="moz-txt-link-rfc2396E" href="mailto:robert.paschedag@web.de"> <robert.paschedag@web.de></a> > Sent: Wednesday, December 7, 2016 11:48 PM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > > > I think you should check your entitlements. But I also don't know, how you > could "manually" connect to the repo (with username/password or token inline > in URL). > > Is your server listed as entitled within your RedHat account? > > ________________________________ > > Von: Daryl Rose > Gesendet:‎06.‎12.‎2016 17:26 > An: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Betreff:Re: [Spacewalk-list] RHEL 6.x repository not syncing from > parentchannel > > I decided to try the sync from the command line. This time I received an > error: > > > > ERROR: Cannot retrieve repository metadata (repomd.xml) for repository: > content_dist_rhel_server_6_6Server_x86_64_os_. Please verify its path and > try again > > > > The path came out of the redhat.repo file that came from Red Hat when I > registered the server. Its the same path that was used on the previous > server. > > > > Any suggestions? > > > > Thank you. > > > > Daryl > > > > ________________________________ > > <a class="moz-txt-link-abbreviated" href="mailto:From:spacewalk-list-bounces@redhat.com"> From:spacewalk-list-bounces@redhat.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:spacewalk-list-bounces@redhat.com"> <spacewalk-list-bounces@redhat.com></a> > on behalf of Daryl Rose <a class="moz-txt-link-rfc2396E" href="mailto:darylrose@outlook.com"> <darylrose@outlook.com></a> > Sent: Monday, December 5, 2016 3:00 PM > To: <a class="moz-txt-link-abbreviated" href="mailto:spacewalk-list@redhat.com"> spacewalk-list@redhat.com</a> > Subject: [Spacewalk-list] RHEL 6.x repository not syncing from parent > channel > > > > I just realized that the last Red Hat errata for RHEL 6.x is from August. I > looked at the sync logs and it doesn't appear that anything is syncing at > all from the RHEL channel. > > > > I just recently migrated to a physical server. I successfully registered > the server and attached an entitlement to the server. If I run "yum update" > on the server, it lists latest updates from the RHEL channel. > > > > This is what I see in the reposync log: > > > > 2016/12/05 14:49:42 -05:00 Command: ['/usr/bin/spacewalk-repo-sync', > '--channel', 'rhel-6-server-rpms', '--type', 'yum'] > > 2016/12/05 14:49:42 -05:00 Sync of channel started. > > 2016/12/05 14:49:42 -05:00 Repo URL: https: > > 2016/12/05 14:49:42 -05:00 Sync of channel completed in 0:00:00. > > > > If I haven't received any syncs since August, then this broke long before I > migrated to the physical server. > > > > Any thoughts? > > > > Daryl > > > > > > > _______________________________________________ > Spacewalk-list mailing list > <a class="moz-txt-link-abbreviated" href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank"> https://www.redhat.com/mailman/listinfo/spacewalk-list</a> _______________________________________________ Spacewalk-list mailing list <a class="moz-txt-link-abbreviated" href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a></div> </div> </div> _______________________________________________ Spacewalk-list mailing list <a class="moz-txt-link-abbreviated" href="mailto:Spacewalk-list@redhat.com"> Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank"> https://www.redhat.com/mailman/listinfo/spacewalk-list</a></div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <fieldset class="mimeAttachmentHeader"></fieldset> <pre>_______________________________________________ Spacewalk-list mailing list <a class="moz-txt-link-abbreviated" href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> <a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/spacewalk-list">https://www.redhat.com/mailman/listinfo/spacewalk-list</a></pre> </blockquote> **** DISCLAIMER **** <a href="http://www.schaubroeck.be/maildisclaimer.htm" style="color:#002E6D">http://www.schaubroeck.be/maildisclaimer.htm</a> </div> </div> </div> <fieldset class="mimeAttachmentHeader"></fieldset> <pre>_______________________________________________ Spacewalk-list mailing list <a class="moz-txt-link-abbreviated" href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> <a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/spacewalk-list">https://www.redhat.com/mailman/listinfo/spacewalk-list</a></pre> </blockquote> **** DISCLAIMER **** <a href="http://www.schaubroeck.be/maildisclaimer.htm" style="color:#002E6D">http://www.schaubroeck.be/maildisclaimer.htm</a> </div> </div> </div> </body> </html>