Serving everything under SSL causes slowdown

I suspect the slowness is purely because we serve everything over SSL.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

I think Stuart's right. It's not so much the intrinsic overhead of SSL, it's that browsers typically don't cache resources served over https. This has the unfortunate result that linked images, css and javascript are redownloaded everytime. At the same time, browsers complain if you try to link to things like images at http URLs from https pages, with big scary warnings that "some parts of this page may be insecure."

The best solution I know of is to only use https for the login form, because that transmits a password, and let everything else be in the clear.

Serving everything over https causes an even greater slowdown on satellite internet connections such as Hughesnet or WildBlue, as the provider-side optimizations used to cope with the high latency don't function at all with https. As a result, Launchpad is slower on satellite than it is on dial-up due to the use of https everywhere - which has been a MAJOR issue for me when testing Ubuntu's unstable distribution (as I have a satellite connection). This in a way discourages me from reporting bugs - and has me thinking of switching to and testing Debian, who doesn't have an HTTPS-only bug tracker.

Is it possible to switch to using https for the password only, or at least offer this as an option? Those of us on high-latency and low-bandwidth connections would especially appreciate it, as the status quo makes using launchpad an unpleasant experience. I'd really like to hear from the devs on this, as I'm effected greatly by this right now.

We are working towards requiring SSL only for the authentication screens, registration screens and a small subset of Launchpad we identify as requiring the extra level of security. We are currently putting in place the infrastructure that this requires.

One question - does the Ubuntu wiki use Launchpad as well?
Because this same issue exists there too...

In any case, I'm glad to hear that this is being taken care of...

On 6/29/07, Stuart Bishop <email address hidden> wrote:
>
> We are working towards requiring SSL only for the authentication
> screens, registration screens and a small subset of Launchpad we
> identify as requiring the extra level of security. We are currently
> putting in place the infrastructure that this requires.
>
> --
> Serving everything under SSL causes slowdown
> https://bugs.launchpad.net/bugs/46591
> You received this bug notification because you are a direct subscriber
> of the bug.
>

I think any specification of when we shouldn't use SSL should take into account why bugzilla.mozilla.org switched to serving everything over SSL. (I don't know why, but I can ask.)

Might it be possible to work with some browser developers on caching CSS/JS/images over SSL? I can imagine that this might not be a good idea in all circumstances, but given that this isn't the only bug tracker which runs over SSL, some kind of header ("X-Always-Cache: yes" or something) or other mechanism for flagging certain SSL-served content as cacheable would show significant benefits for the users, in terms of increased speed, and for site owners, in terms of fewer hits on their static content.

As for the reason bugzilla.mozilla.org switched over, see http://gemal.dk/blog/2004/09/21/bugzillamozillaorg_protected_with_ssl_encryption/. The idea is that some bug reports may contain sensitive information (e.g., non-public security advisories), so the whole shebang is encrypted end-to-end to guard against eavesdroppers. It also protects against eavesdropping on the login process, but one would only need to protect the login pages to fix that.

For Firefox 3, sending "Cache-Control: public" will allow it to cache SSL content to disk with its default settings. Tom has updated staging.launchpad.net to serve the /+icing directory with this HTTP response header and things seem to work fine. We'll probably do the same for the lpnet and edge instances shortly.

The images in staging are well-cached now, thanks!

But the librarian files (project icons, logos, etc.) are still not served with cache-control, however trying to fetch them redirects to http://launchpadlibrarian.net/*

Anyway, it looks like FF3 still requires Cache-control for them, because they are retreived again and again, probably due to the whole page being protected.

We've got Cache-Control headers turned on for staging.launchpadlibrarian.net too now, and bumped the expiry up to a week (for both the librarian and /+icing, the content at a URL should not change). We'll be rolling it out to the main instance soon.

Images in staging don't work at all for me for 3 hours already (not displayed in pages), wget output:

$ LANG=C wget -S https://staging.launchpadlibrarian.net/8693353/logo.png
--12:02:47-- https://staging.launchpadlibrarian.net/8693353/logo.png
           => `logo.png'
Resolving staging.launchpadlibrarian.net... 91.189.90.236
Connecting to staging.launchpadlibrarian.net|91.189.90.236|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 503 Service Temporarily Unavailable
  Date: Tue, 06 May 2008 05:03:00 GMT
  Content-Length: 448
  Connection: close
  Content-Type: text/html; charset=iso-8859-1
12:03:19 ERROR 503: Service Temporarily Unavailable.

This looks like a temporary failure -- I'll get one of the admins to fix it later if it doesn't sort itself out.

Note that the Cache-Control header change for the librarian doesn't make a huge difference on staging though: any file not found in the staging librarian results in a redirect to the production librarian (which doesn't yet send such headers). I've verified that the pages being served directly by the staging librarian do have the right headers.

Did this get into trunk? if so, closable perhaps?

The /+icing/... and /@@/... resources appear to be getting served with an appropriate Cache-Control header. The librarian files don't seem to be.

Caching files from the Librarian is bug 5456.

http://fscked.org/category/tags/defcon includes articles on how sites with HTTPS login and HTTP everything-else are vulnerable to session stealing, and how to prevent it.

@mpt Based on a _brief_ reading of that article, it seems the attack will not work against sites that set the https-only bit on cookies. I've been told by Francis that Launchpad does this and just confirmed it by looking at my browser cookies.

But more generally it's true that allowing unencrypted authenticated connections does increase the attack surface.

See also bug 337517 for a smaller version of this bug.

More discussion on launchpad and some numbers:

http://bugs.launchpad.net/launchpad/+bug/361739

https://lists.launchpad.net/launchpad-users/threads.html#04962

Some numbers of how long it takes....

T1 line:

Launchpad vs google (7x time slower)
launchpad vs sourceforge (3x slower)
launchpad vs bitbucket (6x slower) (Can't compare due to size of hosting)

 time wget https://launchpad.net/datahub
real 0m2.060s
user 0m0.048s
sys 0m0.012s

time wget https://code.launchpad.net/datahub

real 0m0.870s
user 0m0.032s
sys 0m0.012s

time wget https://bugs.launchpad.net/datahub
real 0m3.335s
user 0m0.032s
sys 0m0.004s

time wget https://blueprints.launchpad.net/datahub
real 0m0.821s
user 0m0.052s
sys 0m0.008s

 time wget http://bazaar.launchpad.net/~szybalski/datahub/trunk/files

real 0m4.887s
user 0m0.000s
sys 0m0.008s

 time wget http://code.google.com/p/harvestman-crawler/

real 0m0.282s
user 0m0.004s
sys 0m0.004s

time wget http://code.google.com/p/harvestman-crawler/issues/list

real 0m0.467s
user 0m0.000s
sys 0m0.020s

time wget http://sourceforge.net/projects/uicrawler/
real 0m0.705s
user 0m0.004s
sys 0m0.012s

time wget http://sourceforge.net/tracker/?group_id=178471&atid=885363

real 0m0.637s
user 0m0.008s
sys 0m0.004s

time wget http://bitbucket.org/ianb/pip/src/
real 0m0.332s
user 0m0.012s
sys 0m0.004s

I'm going to close this as wontfix - and I'm opening a separate bug listing the various bits of optimising and tuning we're doing to get good performance out of SSL.

Making the application switch between HTTP and HTTPS on a per-page basis is extremely complex. For instance, a bug search that returns a private bug has to return on HTTPS. But if the search is submitted on HTTP we probably have leaked some info about the bug (e.g. a customer name, product name etc) in the submission. Further, we add a late SSL handshake when we do switch to HTTPS, and the SSL session is going to be likely stale by now.

The users that need the best possible performance - the folk doing massive amounts of bug triage - are often the same users that have the most access to private data and are least able to make use of HTTP.

So the benefits to most of the heavy users of the system would be marginal, and we want to make Launchpad fast for them anyway. If we make it fast for the heaviest users on HTTPS we can make it fast for casual users on HTTPS too with a bit of care.

On 14 January 2011 17:59, Robert Collins <email address hidden> wrote:
> I'm going to close this as wontfix

+1

--
Martin