Provide official Ubuntu metalink files on a public webserver

As per discussion at UDS Boston (https://blueprints.launchpad.net/ubuntu/+spec/metalink-iso-downloads), I have evaluated the following program (http://metamirrors.nl/node/60) to generate metalink files from an ISO and a list of mirrors.

It looks good to me, would it be possible to add that to the CD Image building process and have the resulting files uploaded somewhere within ubuntu.com?

There's also the Metalink Library for PHP and Python ( http://code.google.com/p/metalink-library/ ), in case Python would integrate better. metalink-library is a fork of the Metalink Editor by Hampus Wessman.

Steve, would you have a look at incorporating this at some point?

Of course the metalinks can also be generated by a separate cron job which processes available ISOs, without slowing down the ISO building process directly.

It's been indicated to me that the ideal hosting location for the metalink files, from the wubi POV, is www.ubuntu.com; that requires buy-in from the ubuntu.com webmaster, who has been consulted and is considering the matter.

That would also solve the issue of keeping the mirror list current, since www.ubuntu.com already has this information; but no decision has been made yet that they will be hosted there.

Please also provide MD5SUMS + MD5SUMS.sig for the metalink files
That is required to verify such files.

One last request: the metalink file is generated from a list of mirrors (text file) and an ISO. I would appreciate if the top part of the mirrors list could contain reasonably fast mirrors (possibly 1 or 2 for the main countries). This is simply to go around the limitations of the current Wubi builtin downloader without affecting overall metalink functionality and scope. In particular, there were a lot of complaints in the past about very slow download speeds since the wubi downloader is not smart enough to adjust accordingly (things will change in the future). "Sorting" the mirror list text file (even manually and in a completely unscientific way) would improve the user experience without requiring any code change and without much impact on the server loads (servers would be picked by country first and then randomly).

I didn't notice that the metalink spec supports a "preference" attribute for each mirror. It would be great if the "preference" could be set to something sensible given the mirror performance. Again it does not need to be anything too fancy or scientific, it's sufficient to give a higher preference to a few mirrors with good history.

<url type="ftp" location="us" preference="90">ftp://ftp.example.com/example.ext</url>

I'm not sure that the mirror list source data has any information about which mirrors should be preferred. I can look into this.

I'm also confused by your comment that "servers would be picked by country first and then randomly" -- if this is the case, why should the list be sorted?

Steve, sorry wasn't very clear. Wubi algorithm works as follows:

Mirrors are sorted according to a score. The initial score of each mirror is given by the value in "Preference" + an optional bonus of 50 if the mirror location matches the location which was specified when the downloader was called. Mirrors are then sorted according to the score, and I select a mirror randomly among the servers with the same score. A mirror score goes down when it is selected (to allow trying other mirrors) and if there are download errors.

There is no need to give an accurate score to ALL the mirrors. It is enough to give 50 to most mirrors, give 75-100 to a few mirrors that you know are good, and give 25-0 to a few mirrors that you know are bad.

"among the servers with the highest score"^

The above means that if there are 1 or 2 mirrors per country with score > 50 those will be picked first whenever the country matches, then I will pick among the "foreign" servers with the highest score.

As far as wubi goes, a dead mirror is far better than a slow mirror.... if len(mirrors) > 1

Since of course wubi is only one of the clients that would be using metalinks we shouldn't narrow down the metalink scope because of wubi. Nevertheless I think that the same requirements would be shared by other downloaders. In fact, if metalink are supported, it is a fairly safe guess that the download manager can resume a download and swap mirrors. Hence bandwidth is more important than uptime and reliability. So I'd guess that speed is the main criteria for choosing the "preference" as far as clients go. Of course you might adjust that for load balancing considerations.

None of this changes the fact that we probably don't have any meaningful source data about which mirrors are "good" or "bad" mirrors (which can, in any case, be very subjective based on more than just the country of origin)...

Steve, as mentioned a few times it does not have to be anything scientific. And it is perfectly possible to start with equal weightings and then change it later. Nevertheless location and subjectiveness can account for so much when the bandwidth difference is or order of magnitude... Having 3 to 5 large buckets for preferences (say 75-50-25) would compensate quite well for differences in location/subjectiveness IMO. Particularly since such preferences would mostly be relevant when the local mirrors are out of questions.

At the moment if a Spanish user cannot connect to one of the 2 mirrors there, has equal chances to be connected to releases.ubuntu.com or to a server in a developing country with 64K upload. By assigning equal weightings the users are worse off. Not just that, the developing country server is also worse off, since that gets allocated a share of downloads disproportional to what the server and/or country infrastructure can take.

So far ubuntu mirrors list has been perused mostly by human beings, that can easily adjusts for the lack of information in there, but metalinks are supposed to be read by machines and a bit more guidance is in order. All I am asking, is to pick say 10 servers that you know are "good" (or at least not too bad) to be used as fallbacks... So that in the example above, the third attempt of the Spanish user will hit one of those 10 servers. If you want to get fancy and sample download data in multiple countries to bucket the servers, feel free, but it's not necessary.

I'd guess that assigning a high weighting to the bittorrent listings (for the clients that support it) would also make sense.

"At the moment if a Spanish user cannot connect to either of the 2 mirrors there, he has equal chances ..."

Hi. I'm the creator of the metalink commandline (hashing) utility. If they decide to use a commandline program to create metalinks, I could probably help with some shell-scripting and anything else related to that utility. So feel free to contact me if you need any help with that.

Steve,

Since it is a bit late in the process, it might be worth considering segregating the URL of Wubi-specific metalinks from the URL for public metalinks. In theory Wubi should work well with standard metalinks, but in case problems arise, that will provide some headroom to tweak the Wubi metalinks without affecting other download managers. Such split might also help with deadlines, since there is probably less time pressure for the public metalinks. To begin with http://www.wubi-installer.org/metalinks/8.04/* could be used as a starting point for the Wubi-specific metalinks (they are very basic, such files use a single mirror for daily ISOs and have no md5 info).

first pass of metalink files for daily builds now available:
http://cdimage.ubuntu.com/daily-live/current/

Steve, looks good! I tested with a few metalink clients and everything worked.

Thanks Steve

Tested successfully with Wubi rev 492
Things to tests: partial checksums and mirror selection (using country / preference attributes).

Good to see the first official Ubuntu metalinks!

The XSL stylesheet seems to come from one of my Metalink Library examples for Eclipse back in December.
Unfortunately the Eclipse team did not accept my working version yet, so the URL does not exist.

Please be aware that hardy-desktop-amd64.metalink contains wrong
information: Linux-x86

It will be great if the Metalinks would contain more useful hashes like the partial checksums Agostino mentioned, you can try a quick example with the Metalink Library from the command-line.

Steve, please let me know if there is any change in the metalinks. I would be glad to test them before they are deployed.

On Mon, Apr 21, 2008 at 06:28:36AM -0000, Rene Leonhardt wrote:
> Please be aware that hardy-desktop-amd64.metalink contains wrong
> information: Linux-x86

What is the correct string to put here, please?

hmm looks good here www.metalinker.org/Metalink_3.0_Spec.pdf [4.3.6]
should be Linux-x86 for i386 and Linux-x64 for amd64 (I assume)

Was wrong in Wubi metalinks to begin with.

This is great! Thanks so much! There are 5 minor things that could be done to tweak things.

(1) I'll third the request for partial checksums. These allow downloads to be repaired, which can save users lots of time/bandwidth/frustration instead of restarting a download.

(2) You could serve .metalink files as application/metalink+xml - this will trigger some clients.

(3) I made a wiki page ( https://help.ubuntu.com/community/Metalink ) to match the BitTorrent wiki page linked to on indexes like http://releases.ubuntu.com/8.04/ which currently looks like:

[ ] ubuntu-8.04-desktop-i386.iso 23-Apr-2008 02:03 699M Desktop CD for PC (Intel x86) computers (standard download)
[ ] ubuntu-8.04-desktop-i386.iso.torrent 24-Apr-2008 08:23 28K Desktop CD for PC (Intel x86) computers (BitTorrent download)
[ ] ubuntu-8.04-desktop-i386.list 23-Apr-2008 02:03 3.9K Desktop CD for PC (Intel x86) computers (file listing)
[ ] ubuntu-8.04-desktop-i386.manifest 22-Apr-2008 19:12 31K Desktop CD for PC (Intel x86) computers (contents of live filesystem)
[ ] ubuntu-8.04-desktop-i386.metalink 24-Apr-2008 08:37 12K Ubuntu 8.04 LTS (Hardy Heron)

If it could link to the Metalink wiki page, for people who don't know about metalink downloads yet, that could help. You could also name it filename.extension.metalink to match the filename.extension.torrent naming scheme, but of course that's up to you. :)

[ ] ubuntu-8.04-desktop-i386.metalink 24-Apr-2008 08:37 12K Desktop CD for PC (Intel x86) computers (Metalink download)

(4) Optionally, you can include the URL to the .torrent in the .metalink. This lets hybrid clients like aria2, GetRight, and KGet (in progress) download from FTP/HTTP and BitTorrent at the same time, while the data downloaded from FTP/HTTP is uploaded to the BitTorrent swarm as well. For ubuntu-8.04-desktop-i386.metalink, this would look like:

<url type="bittorrent">http://releases.ubuntu.com/8.04/ubuntu-8.04-desktop-i386.iso.torrent</url>

(5) I noticed the publisher section of the metalink points to ubuntu.org - I think you want ubuntu.com unless the two are related (they don't appear to be).

  <publisher>
    <name>Ubuntu</name>
    <url>http://www.ubuntu.org/</url>
  </publisher>

The primary driver for implementing metalink files has been Wubi compatibility. I don't think we have much incentive to support other metalink clients; indeed, I'm given to understand that a number of metalink clients are rather pathological network users, and will use metalinks to simultaneously download from multiple mirrors, which is not something we want to endorse - folks should be using bittorrent for that instead.

So if there are other changes needed here, please justify them in terms of Wubi requirements. I know that partial checksums are still wanted for Wubi. We should give a better description of the .metalink files in .htaccess, but not one that points to a wiki page that would encourage direct use of other metalink clients.

Steve, many metalinks clients support bittorrent, and it is possible to add torrents to the metalink (with very high priority I would assume). Wubi does not support bittorrent and it will ignore the relative tag. But adding that can only free up resources as far as other clients go (whether we encourage them or not).

Required Wubi-side changes have been committed weeks ago'

hello,

an old OOo metalink http://download.packages.ro/metalink/openoffice/OOo_1_0_2_SolarisIntel_install_tar_gz.metalink
is well displayed on FX or MSIE

latest 8.04 ubuntu metalink as ftp://ftp.free.fr/mirrors/ftp.ubuntu.com/releases/8.04/ubuntu-8.04-alternate-amd64.metalink
gives another displaying on FX and also gives an error on MSIE (spoke
about http://download.eclipse.org/eclipse/metalink.xsl ???)

Steve, if you want to disable multi-source downloads in other clients, you can change '<resources>' to '<resources maxconnections="1">' in the .metalinks, which tells clients to only connect to one URL at a time. (As usual, if a server goes down, the client will connect to the next one on the list).

I've edited the wiki page so it doesn't list other metalink clients.

Thanks, Ant; I've added this to the template for metalink files we generate in the future.