PPAs only publish the latest version of a package in a distro series

Ted, all binaries ever built in the PPA context are available for download directly from LP (librarian) in the PPA page. Although only the latest binary versions are available in the repository. Let me know if mitigates your problem, or do we need to investigate other alternative.
I can foresee that keeping old binary versions available in the repository tends to increase disk-usage a lot and makes gardening more complicated. Comparing `apt-get install foo-bin=1.0` and going to the PPA page and click on a link to install the via gebi, I can't see a big advantage in the former.

In addition, we're implementing multiple PPAs per user as in bug 158570 and blueprint https://blueprints.edge.launchpad.net/soyuz/+spec/soyuz-multiple-ppas

This will allow a user/team to keep as many versions around as they wish to make PPAs for.

On Tue, 2009-01-13 at 19:57 +0000, Celso Providelo wrote:
> Ted, all binaries ever built in the PPA context are available for
> download directly from LP (librarian) in the PPA page. Although
> only the latest binary versions are available in the repository.
> Let me know if mitigates your problem, or do we need to investigate
> other alternative.

Well, see, I didn't know this. But, I think that's part of the problem.
I know how to use synaptic to select which version of a package I'm
using. I think that's the case for most users.

> I can foresee that keeping old binary versions available in the
> repository tends to increase disk-usage a lot and makes gardening
> more complicated. Comparing `apt-get install foo-bin=1.0` and
> going to the PPA page and click on a link to install the via gebi,
> I can't see a big advantage in the former.

I think there are a few reasons this is still important. Most of the
tools that can install single debs reasonably are command line. Users
should never have to use the command line (I'm a big fan of not shipping
gnome-terminal with Ubuntu). Secondly, by using apt and with the signed
PPAs in the future there is less of a chance of man-in-the-middle
attacks (not huge, but not unreasonble).

In general, I think that if we're already storing the binaries, making
them available to APT isn't such a huge cost and it does have a real
benefit to users.

On Tue, 2009-01-13 at 20:08 +0000, Julian Edwards wrote:
> In addition, we're implementing multiple PPAs per user as in bug 158570
> and blueprint https://blueprints.edge.launchpad.net/soyuz/+spec/soyuz-
> multiple-ppas
>
> This will allow a user/team to keep as many versions around as they wish
> to make PPAs for.

Yes, but then users would have to switch their repositories to switch
versions. Seems like unnecessary overhead.

> Yes, but then users would have to switch their repositories to switch
> versions. Seems like unnecessary overhead.

Well, can't they just enable all of them and forget about it?

Ted,

Installing debs looks deadly easy in intrepid. When I click on a deb listed in the PPA page firefox offers to open it with gdebi, which just require a click to install it.

What might be frustrating is to install a older version of a package that is already installed in you system. Gdebi denies it with a readable red warning and then you have to remove it via terminal. If gdebi presented a 'downgrade' button it would be 100 % fine for your case (single debs).

Again, we come back to the initial proposal to solve this problem, optionally extend the period used the for marking publication superseded or even turn it off. Deciding a unit to control it might be tricky, age (keep old publication available in the indexes for 30 days) or number o old versions (keep up to 6 old versions available for each source).

Either way I'm really concerned by allowing users to flood their PPAs pretty quickly, reach the quota limit and start angrily asking for more space.

The latest and greatest is not always installable due to conflicts. For example, for many users the latest and greatest bzr from the bzr PPA is not useful to them until a compatible bzrtools package is released.

I think keeping old versions of packages in PPAs would be good, providing a better environment for beta testing as users can jump between betas and release candidates without constantly needing to add new repositories.

I agree there needs to be some mechanism to keep disk space usage sane. Number of old packages, total size of packages, age of packages could all be used to keep the PPA size down.

The overwhelming majority of people don't need old versions floating about in their PPAs, especially when it eats their quota. For those who do (for example, the bzr team) we're implementing multiple PPAs per person so that each PPA is able to contain different branches of code as desired (to be released in 2 months).

As Celso, says, older versions are available via direct download from Launchpad, so nobody is blocked on that. For the small benefit it would bring to be able to leave the old versions in the code compared to the effort we need to expend to implement it, I can't see it being worthwhile. I more beneficial change would be to make GDebi able to downgrade packages, then clicking .deb links in Launchpad would be a breeze.

Stuart says:
> latest and greatest bzr from the bzr PPA is not useful to them until a compatible bzrtools package is released.

This is a packaging issue, no amount of trickery on the part of PPAs will solve this. You best bet is to subscribe to the latest stable PPA rather than the development one.

So sorry, this is "won't fix" for now.

On Mon, Jan 19, 2009 at 09:58:09AM -0000, Julian Edwards wrote:
> Stuart says:
> > latest and greatest bzr from the bzr PPA is not useful to them until
> > a compatible bzrtools package is released.
>
> This is a packaging issue, no amount of trickery on the part of PPAs
> will solve this.

Really? I had thought that if we did indeed offer a configurable number
of packages to be kept around, the bzr problem would indeed be fixed
without the need for multiple PPAs. Can you elaborate?

I was referring to the fact that they're shipping a bzr that forces the
removal of an older version of bzrtools because there's no compatible,
newer bzrtools available.

Note that we are working with the hypothesis that apt will behave slightly better if we have multiple binary versions available in the repository index when there is an unstable transition. For instance, in the bzr PPA case:

 * bzr_1.2 (requires bzrtools == 1.2) is published
 * bzr_1.1 (requires bzrtools == 1.1) is published
 * bzrtools_1.1 is published

In this scenario apt will install (of keep) bzr_1.1 and bzrtools_1.1 installed with a discreet warning about the fact that bzr_1.2 exist but cannot be installed without removing bzrtools_1.1. However, if bzr_1.1 gets superseded and thus removed from the repository index (current approach), apt will automatically remove bzrtools and install bzr_1.2.

For standalone binaries (inkscape case) it will indeed allow user to jump back and forward through the available versions.

Before digging into implementation details, I'd like to confirm if apt actually behaves as I described or am I dreaming.

On Tue, Jan 27, 2009 at 11:39 PM, Celso Providelo
<email address hidden> wrote:
> Note that we are working with the hypothesis that apt will behave
> slightly better if we have multiple binary versions available in the
> repository index when there is an unstable transition. For instance, in
> the bzr PPA case:
>
> * bzr_1.2 (requires bzrtools == 1.2) is published
> * bzr_1.1 (requires bzrtools == 1.1) is published
> * bzrtools_1.1 is published
>
> In this scenario apt will install (of keep) bzr_1.1 and bzrtools_1.1
> installed with a discreet warning about the fact that bzr_1.2 exist but
> cannot be installed without removing bzrtools_1.1. However, if bzr_1.1
> gets superseded and thus removed from the repository index (current
> approach), apt will automatically remove bzrtools and install bzr_1.2.
>
> For standalone binaries (inkscape case) it will indeed allow user to
> jump back and forward through the available versions.
>
> Before digging into implementation details, I'd like to confirm if apt
> actually behaves as I described or am I dreaming.

Synaptic's 'Mark all Updates' function will remove bzrtools and
install bzr 1.2 even when the bzr 1.1 package still exists (provided
you haven't locked the bzr version).
Update manager will not remove the 1.1 package.

Different tools will behave differently, even if they are just front
ends to apt.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

> The overwhelming majority of people don't need old versions floating about in their PPAs, especially when it eats their quota.

So if I understand correctly, the old versions are kept around forever in the Librarian, but the quota only applies to the files referenced by the archive? That seems kind of odd. They're still costing us disk space, so I think we should certainly make them accessible.

It's true that if the files are still in the librarian they can be installed by hand, but this is a bit unfriendly for people used to using apt, particularly if eg you need to install multiple packages.

@cprov: I think you're correct. But anyhow, regardless of the default behaviour, keeping the old versions accessible lets people force installation or pin a particular version through the apt interface.

@stub: Yes, different tools behave differently. ;-)

For the specific case of bzr I'd like to work out what people use from bzrtools and merge it into bzr core.

Martin,

Thanks for you comments, right the implementation is beneficial in a sense that superseded/old packages could still be handled via Synaptic.

The drawback is that packages that remain published in the repository cost us 2x disk space (librarian + repository disk). We are almost fine with keep old sources in librarian for a long time, but not that happy about duplicating the space consumption for keeping them in the repository as well.

Do you see my point ?

Consider the case where we have package foo 1.0 and foo 1.1.

Foo 1.1 supersedes foo 1.0. Currently, the only way to keep foo 1.0 published in a repo is to copy it to another PPA. This means creating (if not already done) a new team/person for that PPA.

When we have multiple PPAs per user, this will become a lot more convenient. The only issue is that it will eat more of your quota, but for most users we'll increase that so it's not an issue in practice. The end user can enable both repositories and then has two choices to install the superseded package:
1. click on the .deb and download it from the librarian
2. apt-get install foo=1.0

So, I'd like to understand from our users if/why this is a problem? Is there a better way that suits us all?

2009/2/4 Celso Providelo <email address hidden>:
> Martin,
>
> Thanks for you comments, right the implementation is beneficial in a
> sense that superseded/old packages could still be handled via Synaptic.
>
> The drawback is that packages that remain published in the repository
> cost us 2x disk space (librarian + repository disk). We are almost fine
> with keep old sources in librarian for a long time, but not that happy
> about duplicating the space consumption for keeping them in the
> repository as well.
>
> Do you see my point ?

I do. I hadn't realized that the repository disk was separate from
the librarian disk. I though the repository pointed (by http
redirects or rewrites) into the librarian.

I still think keeping old versions for some period of time would be
highly useful, and it could be capped so as not to use too much space.

--
Martin <http://launchpad.net/~mbp/>

2009/2/4 Julian Edwards <email address hidden>:
> Consider the case where we have package foo 1.0 and foo 1.1.
>
> Foo 1.1 supersedes foo 1.0. Currently, the only way to keep foo 1.0
> published in a repo is to copy it to another PPA. This means creating
> (if not already done) a new team/person for that PPA.

When you say "the only way", do you mean "the only way within the
current Soyuz code", or "the only way that works in Debian archives at
all"?

The first may be true but I don't think the second is. Manually
maintained archives are perfectly happy holding multiple versions of a
package and this is IME pretty common for archives maintained by
upstream developers. We always had this for baz and bzr before moving
to PPAs.

--
Martin <http://launchpad.net/~mbp/>

On Wed, Feb 04, 2009 at 09:01:26AM -0000, Martin Pool wrote:
> So if I understand correctly, the old versions are kept around forever
> in the Librarian, but the quota only applies to the files referenced by
> the archive? That seems kind of odd. They're still costing us disk
> space, so I think we should certainly make them accessible.

For the record, we're also working on cleaning out old versions from the
Librarian. So we are indeed concerned with disk space.

Keeping a set of old versions is not a bad idea, having said that. I was
thinking that one way of solving this would be increasing the stay of
execution time to something longer. For how long would you want
this version to hang around, Martin?

Another option would be to change the policy for removing superseded
packages; for instance, always keeping one superseded version of the
package around.

Martin Pool wrote:
> When you say "the only way", do you mean "the only way within the
> current Soyuz code", or "the only way that works in Debian archives at
> all"?
>
> The first may be true but I don't think the second is. Manually
> maintained archives are perfectly happy holding multiple versions of a
> package and this is IME pretty common for archives maintained by
> upstream developers. We always had this for baz and bzr before moving
> to PPAs

Yes, I meant "currently in Soyuz." As you say, it's perfectly feasible
to present multiple versions from the same repo. The reason Soyuz
doesn't keep them around is driven from a distro team requirement, and
we carried that paradigm into PPAs (which is arguably more important so
people don't blow their quota as easily).

I see a couple of orthogonal issues:
1. Some people want the binaries in the librarian to live forever (OEM
services for example)
2. Some people want the binaries in the repo to live almost forever. By
almost, I think you want to be ideally able to say "I don't need this
old version any more" and have it superseded on demand, right?

It would be a very nice piece of work for us to make this all
configurable. In fact, it presents commercial revenue options so I'd
like to do it, but unless Kiko says otherwise I can't see it getting
done before 3.0.

On Thu, Feb 05, 2009 at 12:23:24PM -0000, Julian Edwards wrote:
> Yes, I meant "currently in Soyuz." As you say, it's perfectly feasible
> to present multiple versions from the same repo. The reason Soyuz
> doesn't keep them around is driven from a distro team requirement, and

A distro team requirement? What's that?

Christian Reis wrote:
> A distro team requirement? What's that?
>

AIUI, the Soyuz code was originally written to remove superseded
binaries from the repo because that's what they wanted.

2009/2/5 Christian Reis <email address hidden>:

> Keeping a set of old versions is not a bad idea, having said that. I was
> thinking that one way of solving this would be increasing the stay of
> execution time to something longer. For how long would you want
> this version to hang around, Martin?
>
> Another option would be to change the policy for removing superseded
> packages; for instance, always keeping one superseded version of the
> package around.

Either of those could be good. It sounds like you also want a flag
saying that a particular team is allowed to retain versions
indefinitely, or for a longer period, or more different versions.

--
Martin <http://launchpad.net/~mbp/>

Ok so we have a clear problem in that people need to keep old versions around.

We'll introduce a per-PPA setting that allows the owner to configure the number of versions to keep for old sources before they are superseded, or to keep them indefinitely. This means if you had say, "keep 2 old versions plus the current one" you'd get something like this:

foo 1.0 (SUPERSEDED)
foo 1.1 (PUBLISHED)
foo 1.2 (PUBLISHED)
foo 1.3 (PUBLISHED)

I think this pretty much what the original bug report wanted and there's good justification for it in the comments.

Thanks to all the bug commenters for helping to see the use cases here.

That sounds good.

Just to answer one thing raised before, using multiple PPAs, even per person, would not be very good because you'd presumably need to semi-manually copy the old versions into the other PPAs.

Fixing this would probably implicitly fix bug 209515 for practical purposes, and have other benefits besides, such as Ted's case.

While debugging NM with asac, I needed to get older debs to check for regression.
To do this, I had to navigate thought several pages and then access each of the required (and dependency) debs, then download every single one of them via web browser to finally be able to install them.

Since the debs are already in LP, how can we "make it easy" to access them via APT?

Of course making LP learn about superseed packages, and making an APT/PPA repo where does packages are linked to, allowing an user to add the repo to sources, and PIN Back the version would be an huge step forward. I'm just not sure if its possible.
What do you guys think?

The problem i see with keeping older packages is that PPA fills up way
too fast (at least for mozilla packages)
I suggest maybe not keeping them in the PPA but have the older packages
get stored somewhere else. Problem with this is server space, but it
would be nice to see how this turns out

2009/3/25 John Vivirito <email address hidden>:
> The problem i see with keeping older packages is that PPA fills up way
> too fast (at least for mozilla packages)
> I suggest maybe not keeping them in the PPA but have the older packages
> get stored somewhere else. Problem with this is server space, but it
> would be nice to see how this turns out

It presumably uses the same amount of space whether they're indexed in
the PPA or not. Is there any advantage to keeping them in Launchpad
but not in the PPA?

--
Martin <http://launchpad.net/~mbp/>

There's two places you can use space:

1. In the archive repository.
2. In the librarian.

The former is the one that has a quota, the latter does not. However, the
librarian will expire superseded and deleted binaries in the librarian 30 days
after the files were unpublished (except for our blacklist of PPAs).

So - don't worry too much about how much space you're using. If it's a useful
project to the community you'll generally be granted more space if you ask.