bzr doesn't recognize 404 errors when the server redirects errors to a dedicated page

As indicated in bug #129307, branching from some servers that redirect before reporting a 404 error does not work. However, once the initial branching is done (by other means), merging was working with bzr 1.3.1. With bzr 1.4 (or bzr 1.5dev), that fails saying the URL is not a branch.

To reproduce the bug, create a new directory, untar the attached tarball which will create a .bzr/ directory, and do

  bzr merge http://users.skynet.be/Pierre.Brukier/anum/projet/

This works as it should with bzr 1.3.1 but with the 1.5dev one gets:

$ bzr merge
Merging from remembered location http://users.skynet.be/Pierre.Brukier/anum/projet/
http://users.skynet.be/Pierre.Brukier/anum/projet/ is redirected to http://hostingerrors.isp.belgacom.be/Errors.cgi?url=http://users.skynet.be/Pierre.Brukier/anum/projet/
http://users.skynet.be/Pierre.Brukier/anum/ is redirected to http://hostingerrors.isp.belgacom.be/Errors.cgi?url=http://users.skynet.be/Pierre.Brukier/anum/
http://users.skynet.be/Pierre.Brukier/ is redirected to http://hostingerrors.isp.belgacom.be/Errors.cgi?url=http://users.skynet.be/Pierre.Brukier/
bzr: ERROR: Not a branch: "http://users.skynet.be/Pierre.Brukier/anum/projet/".

Trying to get any non-existing file or to obtain the content of a directory will _redirect_ to a 404 page. This is really annoying as this means collaborating with people using this service provider (which is rather common in Belgium) will be harder that it should.

Presumably following redirections (with a warning on stderr if deemed necessary) and not relying on being able to get the content of directories will fix the problem.

P.S. May be related to #36004