[Regression] RHEL 5.3 yum handles circular dependencies wrong, causes mock failures

Description of problem:
I upgraded our internal build server from RHEL 5.2 to RHEL 5.3 and now I've
got problems with initializing the mock buildroot (installing all dependencies
of buildsys-build fails) with the following error:

--- snipp ---
/var/tmp/rpm-tmp.7553: line 6: cat: command not found
/var/tmp/rpm-tmp.7553: line 7: rm: command not found
/var/tmp/rpm-tmp.7553: line 22: install: command not found
/var/tmp/rpm-tmp.7553: line 25: install: command not found
error: %post(pam-0.99.6.2-4.el5.i386) scriptlet failed, exit status 127
--- snapp ---

It's a circular dependency between pam and coreutils which is wrong solved
by yum in RHEL 5.3 while it was correct solved by yum in RHEL 5.2. Means,
that pam is installed before coreutils, which never should be case. And this
was working in RHEL 5.2 as expected. Somehow, this bug only shows up when
building for mock target i386, the host arch does not matter as it seems.

Version-Release number of selected component (if applicable):
yum-3.2.8-9.el5 (RHEL 5.2, works)
yum-3.2.8-9.el5_2.1 (RHEL 5.3, doesn't work)

How reproducible:
Everytime, see above and below.

Steps to Reproduce:
1. Install a system with RHEL 5.2 (thus yum-3.2.8-9.el5)
2. Enable EPEL and install latest mock package from EPEL
3. mock -r epel-5-i386 rebuild <src.rpm>
4. See it working
5. Update system to RHEL 5.3 (thus yum-3.2.8-9.el5_2.1)
6. mock -r epel-5-i386 rebuild <src.rpm>
7. See it failing with the error above.
8. Downgrade to yum from RHEL 5.2
9. mock -r epel-5-i386 rebuild <src.rpm>
10. See it working again

Actual results:
RHEL 5.3 yum handles circular dependencies wrong, causes mock failures, but
somehow only on i386 mock target architecture, not for x86_64.

Expected results:
RHEL 5.3 yum working same as yum from RHEL 5.2.

Additional info:
http://www.centos.org/modules/newbb/viewtopic.php?topic_id=19771&forum=37&post_id=74283
http://lists.centos.org/pipermail/centos/2009-April/074912.html

Re-read above with s/yum-3.2.8-9.el5_2.1/yum-3.2.19-18.el5/g please, was a
copy & paste error. Without sed:

- yum-3.2.8-9.el5 (RHEL 5.2) works
- yum-3.2.19-18.el5 (RHEL 5.3) does not work

Please verifify and fix ASAP, breaks mock builds for CentOS and RHEL all the
time, unluckily.

 There are at least four "bugs" here:

1. Pam doesn't do the hack to make sure coreutils is installed first.

2. Yum in 5.3 fails the transaction when scriptlets fail.

3. rpm marks %post failures as a scriplet failure.

4. rpm changes it's ordering of pam/coreutils, which without #1 it is entitled to do.

...the way this was "fixed" in Fedora was to have rpm not mark the transaction as a scriptlet failure when %post fails (fixing #3). One way to fix it for 5.3 is to fix pam (I know some people rebuilding CentOS have done this), which takes care of #1. And the other work around is:

http://lists.centos.org/pipermail/centos/2009-April/075092.html

...which fixes #4. Reverting #2, which I assume is what you want given where you opened the bug, is possible ... but the least preferable IMO.
 I'll pass this over to Panu to see if he wants to do #3 for 5.4.

Note that just adding "Requires(post): coreutils, /sbin/ldconfig" to pam does
not work for me, that is IIRC the last suggestion on the CentOS mailing list
thread regarding that.

Interestingly, I've to downgrade yum and only yum, not rpm to get it working
again. So if you tell me, that this is also a rpm issue, yum would duplicate
the logic of rpm and the bug would exist twice - once in yum and once in rpm?

There's a whole pile of dependency loops here, just to name a few:
pam <-> coreutils <-> libselinux <-> findutils <-> coreutils

This makes ordering quite a lottery, for example in this case hitting the pam %post failure depends on the exact order in which the packages are fed into the transaction: apparently that order in yum happened to change between 5.2 and 5.3. And the ordering depends on the exact set of dependencies present in the transaction, another change here is that rpm itself used to have %pre require shadow-utils which in turn depends on libselinux which is involved in the loops above... which might've masked the underlying loops in this case.

The dependency loops are what really needs fixing, but it's possible to work around with adding dependency whiteout to rpm config, this seems to cure at least this particular case:

%_dependency_whiteout \
coreutils>pam

Well, yum from RHEL 5.2 was working. With yum from RHEL 5.3 it is no longer
working, thus it is a regression. Why can't we fix the issue by undo what has
been done to break it?

That yum in 5.2 happens to add things to the transaction in an order that happens to work for this case is just *pure luck*. Topological sort gives different ordering when the input changes, but those orders are correct if the input (dependencies in this case) are correct. The dependencies need fixing, not yum or rpm.

this bug is the same as #497570

*** This bug has been marked as a duplicate of bug 497570 ***