fuellib_review_pkgs_master_node does not use packages built as part of change

Some update, from discussion with Artem:

No aux repo in pinning rules:

root@node-1:~# cat /etc/apt/preferences.d/*
Package: *
Pin: release
Pin-Priority: 1150Package: *
Pin: release o=Mirantis,a=mos8.0-holdback,l=mos8.0,n=mos8.0-holdback
Pin-Priority: 1100Package: *
Pin: release n=mos8.0,o=Mirantis,l=mos8.0,a=mos8.0
Pin-Priority: 1050Package: *
Pin: release o=Mirantis,n=mos8.0-security,a=mos8.0-security,l=mos8.0
Pin-Priority: 1050Package: *
Pin: release a=mos8.0-updates,o=Mirantis,n=mos8.0-updates,l=mos8.0
Pin-Priority: 1050Package: *
Pin: release l=Ubuntu,o=Ubuntu,n=trusty,v=14.04,a=trusty
Pin-Priority: 1001Package: *
Pin: release l=Ubuntu,o=Ubuntu,a=trusty-updates,n=trusty,v=14.04
Pin-Priority: 1001Package: *
Pin: release l=Ubuntu,o=Ubuntu,n=trusty,v=14.04,a=trusty-security

So built packages in aux repo, has default pinning rules:

root@node-1:~# apt-cache policy fuel-misc
fuel-misc:
  Installed: 8.0.0-7288.1.gitacfcfd2
  Candidate: 8.0.0-7288.1.gitacfcfd2
  Version table:
     8.0.0-7292.1.git84cdfdf 0
        500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
 *** 8.0.0-7288.1.gitacfcfd2 0
       1050 http://10.109.5.2:8080/2015.1.0-8.0/ubuntu/x86_64/ mos8.0/main amd64 Packages
        100 /var/lib/dpkg/status

Set to critical, since we have CI impact

Critical for 7.0 since the same issue:

fuel-snapshot-2015-10-09_04-01-08/node-1.test.domain.local/commands/ubuntu_repo_list.txt

[node-1.test.domain.local] out: 1100 http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/ mos7.0-holdback/main amd64 Packages
[node-1.test.domain.local] out: release o=Mirantis,a=mos7.0-holdback,n=mos7.0,l=mos7.0,c=main
[node-1.test.domain.local] out: origin mirror.fuel-infra.org
[node-1.test.domain.local] out: 500 http://10.109.0.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
[node-1.test.domain.local] out: release c=main
[node-1.test.domain.local] out: origin 10.109.0.2
[node-1.test.domain.local] out: Pinned packages:
[node-1.test.domain.local] out:

Ryan's run [0] was analyzed. In agree with snapshot, all three nodes has correct record for aux repo in astute.yaml:
'''
  - name: Auxiliary
    section: main
    uri: http://10.109.5.2:8080/ubuntu/auxiliary/
    priority: 1150
    suite: auxiliary
    type: deb
'''

But after provisioning in /etc/apt/preferences.d/* aux repo has wrong priority:
'''
500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
release c=main
origin 10.109.5.2
'''

Reassigned to fuel-lib for further investigation, because it's all ok on fuel-qa side.

---
[0] https://ci.fuel-infra.org/job/fuellib_review_pkgs_master_node/4775/artifact/logs/4775/fail_error_deploy_neutron_tun-fuel-snapshot-2015-10-02_14-17-34.tar.xz

On my locally deployed env, I see
root@node-2:~# apt-cache policy | grep -A3 aux
1150 http://10.20.0.2:8080/2015.1.0-8.0/ubuntu/auxiliary/ auxiliary/restricted amd64 Packages
     release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=restricted
     origin 10.20.0.2
1150 http://10.20.0.2:8080/2015.1.0-8.0/ubuntu/auxiliary/ auxiliary/main amd64 Packages
     release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=restricted
     origin 10.20.0.2

From that snapshot from the CI job, it's:
[mmosesohn@localhost commands]$ grep -A3 aux ubuntu_repo_list.txt
[node-1.test.domain.local] out: 500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
[node-1.test.domain.local] out: release c=main
[node-1.test.domain.local] out: origin 10.109.5.2

The regenerate-ubuntu-repo script is generating the wrong Release metadata.
currently it has :
release c=main

but it needs to have:
release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=main

The pinning file depends on generating the correct Release metadata

The release file in .fuel-library.pkgs.ubuntu.neutron_vlan_ha (good):
http://paste.openstack.org/show/gNfcy5l6XHbJjkpIxBLg/
The release file in fuellib_review_pkgs_master_node (bad):
http://paste.openstack.org/show/lPN43dqN2dYqnzHiXZAK/

Fix proposed to branch: master
Review: https://review.openstack.org/234304

The issue happened because new packages are uploaded by fuel-qa tests before the Fuel admin node deployed and initialize auxiliary repositories with correct metadata. Thus, for Ubuntu, auxiliary repository gets empty metadata in 'Release' file and cannot be correctly processed.

Reviewed: https://review.openstack.org/234304
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=a8c4492e12ce58badffa5e90caab6672c6c0e732
Submitter: Jenkins
Branch: master

commit a8c4492e12ce58badffa5e90caab6672c6c0e732
Author: Dennis Dmitriev <email address hidden>
Date: Tue Oct 13 18:39:58 2015 +0300

    Separate uploading packages for patching

    - RPM packages are uploaded after Fuel admin node bootstrapped,
      as previous;
    - DEB packages are uploaded after Fuel admin node deployed and
      'auxiliary' repository initialized.

    Change-Id: I5c423529e175ef6ddfc9ac5f5e2359e1e0b6d32f
    Closes-Bug:#1502242

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/238099

Reviewed: https://review.openstack.org/238099
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=e5e5c1ed43cc1d2544cf215d4bd743eb6e165405
Submitter: Jenkins
Branch: stable/7.0

commit e5e5c1ed43cc1d2544cf215d4bd743eb6e165405
Author: Dennis Dmitriev <email address hidden>
Date: Tue Oct 13 18:39:58 2015 +0300

    Separate uploading packages for patching

    - RPM packages are uploaded after Fuel admin node bootstrapped,
      as previous;
    - DEB packages are uploaded after Fuel admin node deployed and
      'auxiliary' repository initialized.

    Change-Id: I5c423529e175ef6ddfc9ac5f5e2359e1e0b6d32f
    Closes-Bug:#1502242
    (cherry picked from commit a8c4492e12ce58badffa5e90caab6672c6c0e732)