<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72"> <div class="WordSection1"> Hello folks,<o:p></o:p> <o:p> </o:p> I am looking at a rather obscure issue related to the way the RPM library behaves on SUSE. While trying to run remediation for a rule related to RPMs, the process hangs until it finally times out.<o:p></o:p> <o:p> </o:p> # oscap xccdf eval --profile test --remediate sles11-xccdf.xml<o:p></o:p> Title Uninstall bind Package<o:p></o:p> Rule package_bind_removed<o:p></o:p> Ident CCE-27030-6<o:p></o:p> Result fail<o:p></o:p> <o:p> </o:p> <o:p> </o:p> --- Starting Remediation ---<o:p></o:p> <o:p> </o:p> Title Uninstall bind Package<o:p></o:p> Rule package_bind_removed<o:p></o:p> Ident CCE-27030-6<o:p></o:p> <o:p> </o:p> <<Hangs for a while here>><o:p></o:p> <o:p> </o:p> Result error<o:p></o:p> <o:p> </o:p> I have collected openscap verbose logs and RPM logs and it looks like there is a deadlock.<o:p></o:p> <o:p> </o:p> From RPM verbose mode logs while the fix is running the “rpm –e” operation:<o:p></o:p> <o:p> </o:p> D: opening db environment /var/lib/rpm/Packages create:cdb:mpool:private<o:p></o:p> D: opening db index /var/lib/rpm/Packages create mode=0x42<o:p></o:p> warning: waiting for exclusive lock on /var/lib/rpm/Packages<o:p></o:p> error: cannot get exclusive lock on /var/lib/rpm/Packages<o:p></o:p> D: closed db index /var/lib/rpm/Packages<o:p></o:p> D: closed db environment /var/lib/rpm/Packages<o:p></o:p> error: cannot open Packages index using db3 - Operation not permitted (1)<o:p></o:p> <o:p> </o:p> Using strace, we can see that it attempts to acquire this WR lock multiple times until it finally times out and returns failure:<o:p></o:p> <o:p> </o:p> fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = -1 EAGAIN (Resource temporarily unavailable)<o:p></o:p> <o:p> </o:p> RPM does not get exclusive mode lock because Openscap seems to be acquiring the lock in read mode:<o:p></o:p> <o:p> </o:p> # lsof /var/lib/rpm/Packages [Command run while the hang is seen]<o:p></o:p> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME<o:p></o:p> probe_rpm 16875 root 3rR REG 8,2 53420032 352263 /var/lib/rpm/Packages<o:p></o:p> rpm 16891 root 3u REG 8,2 53420032 352263 /var/lib/rpm/Packages<o:p></o:p> <o:p> </o:p> # ps –a [Command run while the hang is seen]<o:p></o:p> PID TTY TIME CMD<o:p></o:p> 16859 pts/0 00:00:00 oscap<o:p></o:p> 16865 pts/0 00:00:00 probe_system_in<o:p></o:p> 16870 pts/0 00:00:00 probe_family<o:p></o:p> 16875 pts/0 00:00:00 probe_rpminfo<o:p></o:p> 16885 pts/0 00:00:00 probe_system_in<o:p></o:p> 16890 pts/0 00:00:00 bash<o:p></o:p> 16891 pts/0 00:00:00 rpm<o:p></o:p> 16934 pts/2 00:00:00 ps<o:p></o:p> #<o:p></o:p> <o:p> </o:p> The RPM library call made in function src/OVAL/probes/unix/linux/rpminfo.c:probe_init, rpmtsCreate() and the subsequent rpmtsInitIterator() result in a read lock being taken on the file /var/lib/rpm/Packages.<o:p></o:p> This read lock seems to be released when rpmtsFree() call is made from probe_fini.<o:p></o:p> <o:p> </o:p> The particular probe_rpminfo in question is not related to the OVAL check of the rule but related to the CPE platform check.<o:p></o:p> <o:p> </o:p> From the additional traces I added to openscap:<o:p></o:p> D: probe_rpminfo: ("seap.msg" ":id" 0 (("rpminfo_object" ":id" "oval:org.open-scap.cpe.sles-release:obj:1" ":oval_version" "5.10.1" ) (("name" ":operation" 5 ":var_check" 1 ) "sles-release" ) ) ) [probe_rpminfo(7045):Thream Name(7f901b36c700):seap-packet.c:904:SEAP_packet_recv]<o:p></o:p> …<o:p></o:p> I: probe_rpminfo: gautam : probe_init: Init 1 rpmts [probe_rpminfo(7050):Thream Name(7f45fcd527c0):rpminfo.c:325:probe_init]<o:p></o:p> <<Fix runs between the rpmts create and free .i.e lock acquire and release>><o:p></o:p> I: probe_rpminfo: gautam : probe_fini: Free 1 rpmts [probe_rpminfo(7045):Thream Name(7f90215a37c0):rpminfo.c:343:probe_fini]<o:p></o:p> <o:p> </o:p> Probe_fini is called in this thread only upon receiving SIGTERM.<o:p></o:p> <o:p> </o:p> Why does this work on RHEL?<o:p></o:p> On RHEL, the implementation of the RPM command does not seem to be trying to acquire an exclusive mode lock, it also locks in read mode only:<o:p></o:p> <o:p> </o:p> D: opening db environment /var/lib/rpm cdb:mpool:joinenv<o:p></o:p> D: opening db index /var/lib/rpm/Packages create mode=0x42<o:p></o:p> D: sanity checking 1 elements >> No message about lock acquisition here!<o:p></o:p> D: running pre-transaction scripts<o:p></o:p> <o:p> </o:p> # lsof /var/lib/rpm/Packages <o:p></o:p> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME<o:p></o:p> probe_rpm 33539 root 3rR REG 8,3 75472896 1048584 /var/lib/rpm/Packages<o:p></o:p> rpm 33555 root 3uR REG 8,3 75472896 1048584 /var/lib/rpm/Packages<o:p></o:p> <o:p> </o:p> Is there some design document that explains how openscap spawns the CPE platform check? I am trying to understand if the probe can relinquish the lock by freeing the RPMTs structure before executing the remediation script.<o:p></o:p> <o:p> </o:p> Thank you.<o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Gautam.<o:p></o:p> </div> </body> </html>