Launchpad itself

Allow building livefses against a view of the archive at a fixed point in time

Bug #1765933 reported by Colin Watson on 2018-04-21

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	Launchpad itself	In Progress	High	Colin Watson
	livecd-rootfs (Ubuntu)	Confirmed	Undecided	Unassigned

Bug Description

It would be useful to be able to dispatch a set of livefs builds that all reliably build from the same point in time in the archive's history, even if they're dispatched at slightly different times. We did the InRelease-by-hash work (https://bugs.debian.org/886745, and subsequent Launchpad publisher changes) as a prerequisite for this. We're now ready to consider the next stage. I believe that there are three major chunks of work here:

1) Somebody needs to work out the livecd-rootfs/live-build bits, which I think ought to be a Foundations job (I don't expect to do this part myself). It needs to pick up the inrelease-path option from the sources.list in the environment where livecd-rootfs is run and propagate that through while building the image, but make sure that the inrelease-path option *doesn't* end up in the sources.list in the final image (just like the way we build from ftpmaster.internal but don't actually leave that in the final image, since it wouldn't work).

2) In parallel with 1), we need to improve Launchpad's internal tracking of the history of archive index files. At the moment, the only feasible interface we could offer would be that the entity scheduling livefs builds would need to work out the SHA-256 hashes of all the relevant InRelease files and tell Launchpad what to use. This is cumbersome and hard to automate. Instead, it would be much better to make it work by timestamp (perhaps even simply having an option to pick whatever's current when the builds are scheduled), but to do that we need to convert ArchiveFile into a history table, adding date_created and date_superseded columns so that we can work out which version was live at any given time.

3) Once both 1) and 2) are done, we can add the relevant options to Launchpad's livefs build code to cause the correct sources.list to be dispatched.

Tags:

Related branches

lp://staging/~cjwatson/launchpad/db-archive-file-history

Rejected for merging into lp://staging/launchpad/db-devel

Stuart Bishop: Pending (db) requested 2018-04-21

Launchpad code reviewers: Pending (db) requested 2018-04-21

lp://staging/~cjwatson/launchpad/archive-file-history

Rejected for merging into lp://staging/launchpad

Launchpad code reviewers: Pending requested 2018-04-21

~cjwatson/launchpad:db-archive-file-date-indexes

Merged into launchpad:master

William Grant (community): Approve (db) on 2023-02-01

~cjwatson/launchpad:db-archive-file-date-indexes

Superseded for merging into launchpad:db-devel

William Grant (community): Needs Information (db) on 2023-01-31

~cjwatson/launchpad:archive-file-history-backfill

Merged into launchpad:master

Colin Watson (community): Approve on 2023-01-04

Ioana Lasc: Pending requested 2023-01-04

~cjwatson/launchpad:archive-file-history

Merged into launchpad:master

William Grant (community): Approve (code) on 2022-12-15

~wgrant/launchpad:archive-file-history-refactor

On hold for merging into launchpad:master

Launchpad code reviewers: Pending requested 2020-09-16

~cjwatson/launchpad:archive-file-history-backfill

Superseded for merging into launchpad:master

Ioana Lasc (community): Approve on 2020-09-30

~cjwatson/launchpad:merge-db-stable

Merged into launchpad:master

Colin Watson (community): Approve on 2020-04-10

~cjwatson/launchpad:archive-file-history

Superseded for merging into launchpad:master

Launchpad code reviewers: Pending requested 2019-10-10

~cjwatson/launchpad:db-archive-file-history

Merged into launchpad:db-devel

William Grant (community): Approve (db) on 2020-03-30

Colin Watson (cjwatson) on 2018-04-21

Changed in launchpad:
status:	Triaged → In Progress
assignee:	nobody → Colin Watson (cjwatson)

Revision history for this message

Launchpad Janitor (janitor) wrote on 2018-04-21:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in livecd-rootfs (Ubuntu):
status:	New → Confirmed

Francis Ginther (fginther) on 2018-10-05

tags:

added: id-5b296c1ea29e35b30684caed

Francis Ginther (fginther) on 2020-03-13

tags:

added: id-5e5451e0091a4c48a5bf3c57

Francis Ginther (fginther) on 2020-03-19

tags:

added: id-5e724267b2a8370b605b1367

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2020-10-02:

Is the status of this bug accurate? across xenial..focal spectrum?

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2020-10-02:

so apt in xenial doesn't have support for inrelease-path, which i guess is pre-requisite to build xenial images at point-in-time.

but other series have that, and can be used.

Revision history for this message

Colin Watson (cjwatson) wrote on 2020-10-02:

The status of this bug is accurate. Please see the linked merge proposals for more information; the current blocker is me dealing with a complex set of review feedback on https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/373971. After that, it will still remain for somebody to work out how to integrate this usefully into livecd-rootfs.

As far as I know, nobody has yet worked out how to eliminate magic-proxy entirely. What the pending Launchpad work will give you is a mechanism to query Launchpad for the state of a given metadata file in a given suite at a given point in time, which provides enough information to start the process. However, until PS5 exists (and a significant amount of development work on top of that) we won't actually be able to provide snapshot archives that are directly usable by apt, so livecd-rootfs is still going to have to do a fair amount of massaging of live-build, and somebody needs to dig into the exact details there.

Ayub (zeezo0) on 2020-10-06

information type:	Public → Public Security
information type:	Public Security → Public
information type:	Public → Private Security

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2020-10-07:

Ayub, please do not change the status of bug reports. Thanks.

information type:

Private Security → Public

Zahid Hasan (akkim31-deactivatedaccount) on 2020-11-24

information type:

Public → Private Security

Colin Watson (cjwatson) on 2020-11-24

information type:

Private Security → Public

Bryant Keith Upton (keithupton7496) on 2020-12-29

Changed in livecd-rootfs (Ubuntu):
assignee:	nobody → Bryant Keith Upton (keithupton7496)

Steve Langasek (vorlon) on 2020-12-29

Changed in livecd-rootfs (Ubuntu):
assignee:	Bryant Keith Upton (keithupton7496) → nobody

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2021-03-05:

@colin watson

After fighting with magicproxy & iptables issues. again. I am interested in getting magic proxy doing more-or-less things that might one day make things "nice".

I.e.
make magic proxy, talk to launchpad proxy to access authenticated archives without explicit username/password.

drop iptables nat rules, and instead try to simply export proxy for apt to use.

try to drop proxy, and instead make magic-proxy write out in-release-path stanzas for apt to use based on the passed in timestamp, and make livebuild / debootstrap use all that.

eventually "just" accept the mapping of archive/suites = hash as collected via launchpad api before dispatching the build.

Hopefully above things will make certain things easier as we go along.

Alternatively, maybe we could merge magic-proxy into the proxy that launchpad exports for the builds? I guess probably not, cause all of that is quite hackish.