Comment 6 for bug 913235

in terms of server load or server resources it doesn't really matter - but if it cannot use rsync to scan a mirror if it carries the files, mirrorbrain has to parse the returned html & has to manually iterate over all directories. So if you got hundreds of directories, hundreds of directory listings have to be parsed.

And while mirrorbrain knows about the "big" webserver implementations (apache, etc), there might be modified directory listings in use by some of your mirrors that could make mirrorbrain's html-parsing fail (and thus you might have to tweak mirrorbrain a little to accommodate for those mirrors)