<div dir="ltr">(After setting up a few Spacewalk instances over the years, I've run into one of those... seemingly truly weird issues. Hopefully this is an easy one...)<div><br></div><div>I've got a spacewalk server named spacewalk1 (version 2.4) that had the postgresql DB on it. I recently moved the DB over to another host, and updated /etc/rhn/rhn.conf on the Spacewalk server to point to it.</div><div><br></div><div>Things seemed to work, and then...</div><div><br></div><div>a) I started seeing some newly-registered hosts try to do a 'rhn-profile-sync' and fail with the following:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><p class=""><span class="">Updating package profile...</span></p></div><div><p class=""><span class="">Updating package profile</span></p></div><div><p class=""><span class="">D: rpcServer: Calling XMLRPC registration.welcome_message</span></p></div><div><p class=""><span class="">D: rpcServer: Calling XMLRPC registration.update_packages</span></p></div><div><p class=""><span class="">A protocol error occurred: Internal Server Error , attempt #1,</span></p></div><div><p class=""><span class="">An error has occurred:</span></p></div><div><p class=""><span class="">rhn-plugin: Error communicating with server. The message was:</span></p></div><div><p class=""><span class="">Internal Server Error</span></p></div><div><p class=""><span class="">See /var/log/up2date for more information</span></p></div><div><p class=""><span class=""><br></span></p></div></blockquote><div>
<p class="">Looking on the db server at postgresql-Fri.log (since today is Friday) I see the following:</p></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><p class=""><span class="">2016-02-12 14:15:13.479 CST ERROR: current transaction is aborted, commands ignored until end of transaction block</span></p></div><div><p class=""><span class="">2016-02-12 14:15:13.479 CST STATEMENT: SELECT queue_server(1000011758, 0)</span></p></div><div><p class=""><span class="">2016-02-12 14:15:17.492 CST ERROR: password is required</span></p></div><div><p class=""><span class="">2016-02-12 14:15:17.492 CST DETAIL: Non-superusers must provide a password in the connection string.</span></p></div><div><p class=""><span class="">2016-02-12 14:15:17.492 CST CONTEXT: SQL statement "SELECT dblink_connect('at_conn', 'dbname=' || current_database() || ' port=' || coalesce(inet_server_port(), '5432'))"</span></p></div><div><p class=""><span class="">PL/pgSQL function "pg_dblink_exec" line 5 at PERFORM</span></p></div><div><p class=""><span class="">SQL statement "SELECT pg_dblink_exec( 'insert into rhnPackageEVR(id, epoch, version, release, evr) values (' || $1 || ', ' || $2 || ', ' || $3 || ', ' || $4 || ', evr_t(' || $2 || ', ' || $3 || ', ' || $4 || '))' )"</span></p></div><div><p class=""><span class="">PL/pgSQL function "lookup_evr" line 18 at PERFORM</span></p></div><div><p class=""><span class="">2016-02-12 14:15:17.492 CST STATEMENT: </span></p></div><div><p class=""><span class=""> insert into rhnServerPackage</span></p></div><div><p class=""><span class=""> (server_id, name_id, evr_id, package_arch_id, installtime)</span></p></div><div><p class=""><span class=""> values (1000011758, LOOKUP_PACKAGE_NAME(E'druid'), LOOKUP_EVR(NULL, E'0.8.3', E'0.1'),</span></p></div><div><p class=""><span class=""> LOOKUP_PACKAGE_ARCH(E'noarch'), TO_TIMESTAMP(E'2016-02-11 15:12:08', 'YYYY-MM-DD HH24:MI:SS')</span></p></div><div><p class=""><span class=""> )</span></p></div><div><p class=""><span class=""> </span></p></div><div><p class=""><span class="">2016-02-12 14:15:17.495 CST ERROR: current transaction is aborted, commands ignored until end of transaction block</span></p></div><div><p class=""><span class="">2016-02-12 14:15:17.495 CST STATEMENT: SELECT queue_server(1000011758, 0)</span></p></div></blockquote>b) I'm now seeing the following in the repo sync logs on the spacewalk server (/var/log/rhn/reposync/)<div><p class="">
</p></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>
<p class=""><span class="">[root@spacewalk1 reposync]# tail rhel7-x86_64-ulyaoth.log</span></p>
<p class=""><span class="">Repo URL: <a href="https://repos.ulyaoth.net/RHEL/7/x86_64/os/">https://repos.ulyaoth.net/RHEL/7/x86_64/os/</a></span></p>
<p class=""><span class="">Packages in repo: 436</span></p>
<p class=""><span class="">Packages already synced: 435</span></p>
<p class=""><span class="">Packages to sync: 1</span></p>
<p class=""><span class="">1/1 : ulyaoth-nginx-mainline-1.9.11-1.el7-1.x86_64</span></p>
<p class=""><span class="">(99999, 'ERROR: password is required', 'DETAIL: Non-superusers must provide a password in the connection string.\nCONTEXT: SQL statement "SELECT dblink_connect(\'at_conn\', \'dbname=\' || current_database() || \' port=\' || coalesce(inet_server_port(), \'5432\'))"\nPL/pgSQL function "pg_dblink_exec" line 5 at PERFORM\nSQL statement "SELECT pg_dblink_exec( \'insert into rhnPackageEVR(id, epoch, version, release, evr) values (\' || $1 || \', \' || $2 || \', \' || $3 || \', \' || $4 || \', evr_t(\' || $2 || \', \' || $3 || \', \' || $4 || \'))\' )"\nPL/pgSQL function "lookup_evr" line 18 at PERFORM\n', InternalError('password is required\nDETAIL: Non-superusers must provide a password in the connection string.\nCONTEXT: SQL statement "SELECT dblink_connect(\'at_conn\', \'dbname=\' || current_database() || \' port=\' || coalesce(inet_server_port(), \'5432\'))"\nPL/pgSQL function "pg_dblink_exec" line 5 at PERFORM\nSQL statement "SELECT pg_dblink_exec( \'insert into rhnPackageEVR(id, epoch, version, release, evr) values (\' || $1 || \', \' || $2 || \', \' || $3 || \', \' || $4 || \', evr_t(\' || $2 || \', \' || $3 || \', \' || $4 || \'))\' )"\nPL/pgSQL function "lookup_evr" line 18 at PERFORM\n',))</span></p>
<p class=""><span class="">Linking packages to channel.</span></p>
<p class=""><span class="">Repo <a href="https://repos.ulyaoth.net/RHEL/7/x86_64/os/">https://repos.ulyaoth.net/RHEL/7/x86_64/os/</a> has 0 errata.</span></p>
<p class=""><span class="">Sync completed.</span></p>
<p class=""><span class="">Total time: 0:01:38</span></p>
<p class=""><span class="">[root@spacewalk1 reposync]#</span></p></div></blockquote>It seems like something didn't get updated to tell Spacewalk to use the remote dbusername and password, even though it's hitting the database server.</div><div><br></div><div>Any ideas what I'm missing? Help!</div><div><br></div><div>Thanks,</div><div> -Ian</div></div>