<div dir="ltr">Found the issue. Because these are VMs, I had the DB on a separate disk and moved the DB disk over to the new instance (spacewalk-db1). Turns out that the db user needed to have superuser privs in the DB to make it work. So, one "ALTER USER spacewalkuser with SUPERUSER;" later...<div><br></div><div>I believe this also could have been postgresql ACL-related, but I didn't want to dive too deeply into that as... scary. ;)</div><div><br></div><div> -I</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 12, 2016 at 12:23 PM, Ian Forde <span dir="ltr"><<a href="mailto:ianforde@gmail.com" target="_blank">ianforde@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">(After setting up a few Spacewalk instances over the years, I've run into one of those... seemingly truly weird issues. Hopefully this is an easy one...)<div><br></div><div>I've got a spacewalk server named spacewalk1 (version 2.4) that had the postgresql DB on it. I recently moved the DB over to another host, and updated /etc/rhn/rhn.conf on the Spacewalk server to point to it.</div><div><br></div><div>Things seemed to work, and then...</div><div><br></div><div>a) I started seeing some newly-registered hosts try to do a 'rhn-profile-sync' and fail with the following:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><p><span>Updating package profile...</span></p></div><div><p><span>Updating package profile</span></p></div><div><p><span>D: rpcServer: Calling XMLRPC registration.welcome_message</span></p></div><div><p><span>D: rpcServer: Calling XMLRPC registration.update_packages</span></p></div><div><p><span>A protocol error occurred: Internal Server Error , attempt #1,</span></p></div><div><p><span>An error has occurred:</span></p></div><div><p><span>rhn-plugin: Error communicating with server. The message was:</span></p></div><div><p><span>Internal Server Error</span></p></div><div><p><span>See /var/log/up2date for more information</span></p></div><div><p><span><br></span></p></div></blockquote><div>
<p>Looking on the db server at postgresql-Fri.log (since today is Friday) I see the following:</p></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><p><span>2016-02-12 14:15:13.479 CST ERROR: current transaction is aborted, commands ignored until end of transaction block</span></p></div><div><p><span>2016-02-12 14:15:13.479 CST STATEMENT: SELECT queue_server(1000011758, 0)</span></p></div><div><p><span>2016-02-12 14:15:17.492 CST ERROR: password is required</span></p></div><div><p><span>2016-02-12 14:15:17.492 CST DETAIL: Non-superusers must provide a password in the connection string.</span></p></div><div><p><span>2016-02-12 14:15:17.492 CST CONTEXT: SQL statement "SELECT dblink_connect('at_conn', 'dbname=' || current_database() || ' port=' || coalesce(inet_server_port(), '5432'))"</span></p></div><div><p><span>PL/pgSQL function "pg_dblink_exec" line 5 at PERFORM</span></p></div><div><p><span>SQL statement "SELECT pg_dblink_exec( 'insert into rhnPackageEVR(id, epoch, version, release, evr) values (' || $1 || ', ' || $2 || ', ' || $3 || ', ' || $4 || ', evr_t(' || $2 || ', ' || $3 || ', ' || $4 || '))' )"</span></p></div><div><p><span>PL/pgSQL function "lookup_evr" line 18 at PERFORM</span></p></div><div><p><span>2016-02-12 14:15:17.492 CST STATEMENT: </span></p></div><div><p><span> insert into rhnServerPackage</span></p></div><div><p><span> (server_id, name_id, evr_id, package_arch_id, installtime)</span></p></div><div><p><span> values (1000011758, LOOKUP_PACKAGE_NAME(E'druid'), LOOKUP_EVR(NULL, E'0.8.3', E'0.1'),</span></p></div><div><p><span> LOOKUP_PACKAGE_ARCH(E'noarch'), TO_TIMESTAMP(E'2016-02-11 15:12:08', 'YYYY-MM-DD HH24:MI:SS')</span></p></div><div><p><span> )</span></p></div><div><p><span> </span></p></div><div><p><span>2016-02-12 14:15:17.495 CST ERROR: current transaction is aborted, commands ignored until end of transaction block</span></p></div><div><p><span>2016-02-12 14:15:17.495 CST STATEMENT: SELECT queue_server(1000011758, 0)</span></p></div></blockquote>b) I'm now seeing the following in the repo sync logs on the spacewalk server (/var/log/rhn/reposync/)<div><p>
</p></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>
<p><span>[root@spacewalk1 reposync]# tail rhel7-x86_64-ulyaoth.log</span></p>
<p><span>Repo URL: <a href="https://repos.ulyaoth.net/RHEL/7/x86_64/os/" target="_blank">https://repos.ulyaoth.net/RHEL/7/x86_64/os/</a></span></p>
<p><span>Packages in repo: 436</span></p>
<p><span>Packages already synced: 435</span></p>
<p><span>Packages to sync: 1</span></p>
<p><span>1/1 : ulyaoth-nginx-mainline-1.9.11-1.el7-1.x86_64</span></p>
<p><span>(99999, 'ERROR: password is required', 'DETAIL: Non-superusers must provide a password in the connection string.\nCONTEXT: SQL statement "SELECT dblink_connect(\'at_conn\', \'dbname=\' || current_database() || \' port=\' || coalesce(inet_server_port(), \'5432\'))"\nPL/pgSQL function "pg_dblink_exec" line 5 at PERFORM\nSQL statement "SELECT pg_dblink_exec( \'insert into rhnPackageEVR(id, epoch, version, release, evr) values (\' || $1 || \', \' || $2 || \', \' || $3 || \', \' || $4 || \', evr_t(\' || $2 || \', \' || $3 || \', \' || $4 || \'))\' )"\nPL/pgSQL function "lookup_evr" line 18 at PERFORM\n', InternalError('password is required\nDETAIL: Non-superusers must provide a password in the connection string.\nCONTEXT: SQL statement "SELECT dblink_connect(\'at_conn\', \'dbname=\' || current_database() || \' port=\' || coalesce(inet_server_port(), \'5432\'))"\nPL/pgSQL function "pg_dblink_exec" line 5 at PERFORM\nSQL statement "SELECT pg_dblink_exec( \'insert into rhnPackageEVR(id, epoch, version, release, evr) values (\' || $1 || \', \' || $2 || \', \' || $3 || \', \' || $4 || \', evr_t(\' || $2 || \', \' || $3 || \', \' || $4 || \'))\' )"\nPL/pgSQL function "lookup_evr" line 18 at PERFORM\n',))</span></p>
<p><span>Linking packages to channel.</span></p>
<p><span>Repo <a href="https://repos.ulyaoth.net/RHEL/7/x86_64/os/" target="_blank">https://repos.ulyaoth.net/RHEL/7/x86_64/os/</a> has 0 errata.</span></p>
<p><span>Sync completed.</span></p>
<p><span>Total time: 0:01:38</span></p>
<p><span>[root@spacewalk1 reposync]#</span></p></div></blockquote>It seems like something didn't get updated to tell Spacewalk to use the remote dbusername and password, even though it's hitting the database server.</div><div><br></div><div>Any ideas what I'm missing? Help!</div><div><br></div><div>Thanks,</div><div> -Ian</div></div>
</blockquote></div><br></div>