provide and support a top-level library package for libmozjs (Was: Unable to use libmozjs.so in an application, because of library path problem.)

Ubuntu wants to ship libmozjs as a system library. I think the main thing they want is just for us to put a version number in the libmozjs.so filename.

It seems to me that we should be able to do that on Unix-y platforms. I think I'll just try and do it and ask Jim the questions as they arise. (Jim, it would be helpful if you could hop on IRC and ping me though, so I can do it while you're available to answer my dumb questions.)

We should not do this for 1.9.2, and should do it for 1.9.3 (on all platforms I think... no reason for Windows to be different from unixy platforms in this respect).

Slightly related: bug 469654 will probably change js3250.dll -> js.dll on Windows.

The main issue here is one of policy.

In short, people developing apps for Ubuntu want SpiderMonkey to grow up and start acting like a real library. That means we commit to keeping libmozjs's ABI backward compatible within each stable release branch (such as mozilla-1.9.1).

Either these users would be a net win for us or a net distraction. I encourage discussion on this point. On the whole I think diverse use of libmozjs is a net win, potentially a huge win.

What is the cost of having such a compatibility policy? Two things.

1. Mainly, everyone on the JS team would just need to be aware of it when writing patches that might be backported to a stable branch. Branch owners/release drivers should agree to the policy too. ABI backward compatibility means we can't delete a "friend API" function or change its signature or semantics incompatibly in a stable branch.

2. Second, there's the cost of actually doing releases, which seems small but deserves some consideration given what happened with 1.8 (in short, with tremendous effort I managed to shove a release candidate out the door, but never managed a final release). I think we can minimize this cost just by saying "whatever ships in gecko is an official SpiderMonkey release".

Please comment.

As someone who has installed SpiderMonkey separately (because of CouchDB), it would be very nice if SpiderMonkey acted (more) like a real library.

We're already changing API incompatibly:

http://bit.ly/3mJBqR

Other possible changes include OOM handling, but this one looks like it won't change for the API users:

http://bit.ly/1AjSUj

We definitely want to change the GC rooting API, and add a write barrier:

https://wiki.mozilla.org/JavaScript:SpiderMonkey:GC_Futures

We have had a pretty stable JS API for over a decade, with few (hardly any -- I am having a hard time recalling the last one) compatibility breaks, only some additions. But now we are in a tight competitive race and old API design flaws are restricting how quickly we can optimize.

We could #ifdef JS_BETTER_GC_API, etc. but I would prefer to ifdef the old stuff we're trying to get rid of. It happens that mark/sweep global GC will be needed anyway for the old generation, or the worst case, for any future copying generational GC. So we can probably support bad old API at some not too high cost. In this case!

In general, I'd rather not pay the price any longer. We've done it, it paid in ways we can't measure, there are "SpiderMonkey dark matter" embeddings all over the place (tellme.com and danger.com, now part of Microsoft!), but they are all way downrev AFAICT.

In light of all this, is now the time to be conservative with the JS API? I think not.

/be

(In reply to comment #5)
> We're already changing API incompatibly:
>
> http://bit.ly/3mJBqR
>
> Other possible changes include OOM handling, but this one looks like it won't
> change for the API users:
>
> http://bit.ly/1AjSUj
>
> We definitely want to change the GC rooting API, and add a write barrier:
>
> https://wiki.mozilla.org/JavaScript:SpiderMonkey:GC_Futures

All these things happened or will happen in trunk. I don't think any of them are likely to be backported to a stable branch.

If we ever do want to backport a breaking change to a stable branch, we could just bump the major version number. But that has not happened much in the past and I don't expect it to happen much in the future.

Ok, if we think we can manage major version changes and not change them so fast the downstreams object, then the only issue is the cost of maintaining the stable versions. We don't currently do much to maintain released spidermonkey tarballs. What expectations do system library users bring for maintenance?

/be

Whatever those expectations are, they bring them to their distributors, which in this case is Ubuntu/Canonical. I'm more than fine leaving the packagers to decide how/what to backport as needed, though our experience with embedders indicates that most likely people will just stay downrev and then update their app when they want fixes or features from the new version.

Sounds like a free lunch -- why don't I trust those?

/be

It's not a free lunch, it's a lunch that Ubuntu can buy at a reasonable price if they want to ship the library with different support/backport provisions than upstream will provide. I don't think that we should do backport or other multi-stream support work here. If it's not important enough to Ubuntu's customers for _Ubuntu_ to do it, I don't know why it would be important enough to us!

(In reply to comment #10)
> It's not a free lunch, it's a lunch that Ubuntu can buy at a reasonable price
> if they want to ship the library with different support/backport provisions
> than upstream will provide.

This begs the question of whose reputation gets damaged if the stable version is not well-maintained. We don't have resources to keep a stable version patched for security bugs, perhaps Ubuntu doesn't either -- then what?

/be

I don't think it begs that question, but it does raise it. It's not something we can do anything about -- we can't prevent people from shipping our code as part of their distribution, obviously, so unless we want to grant everyone in the world the right to allocate our time to their pet fixes by simply packaging up our work, I think we're best to just ignore this situation entirely, TBH. In the unlikely event that it causes a problem with noise in the community or press, we should simply point them to this bug, in which we indicated that we weren't going to backport fixes to standalone library, and so people shouldn't distribute it unless they're comfortable with that situation.

(In reply to comment #12)
> I don't think it begs that question, but it does raise it.

Ok, "raises" not "begs". But what is the answer?

> It's not something
> we can do anything about -- we can't prevent people from shipping our code as
> part of their distribution, obviously, so unless we want to grant everyone in
> the world the right to allocate our time to their pet fixes by simply packaging
> up our work,

This is off the mark again, no one is proposing that we enslave ourselves that way.

The issue is what happens if we fix this bug (marking it FIXED) so Ubuntu can and does provide a stable .so as a system library. Sure, they could ship a .so anyway (some do already, IIRC).

But we have a choice: fix this bug, or WONTFIX it (or stall, pretty much the same thing).

> I think we're best to just ignore this situation entirely, TBH.

Do you mean we should fix this bug and ignore the added risk of bad outcomes in the stable .so damaging our rep? Or WONTFIX/stall?

If this bug doesn't matter, then it's INVALID. I think it does matter. The way it matters is that we increase the good rep effects of a useful and safe-enough system lib, and the bad rep effects of the opposite kind of outcome, based on our supporting ("Make it possible") the system library with stable version.

It would be cool to have done this already for the years when our ABI was very stable. Now that we are destabilizing it, I'm a bit concerned about (a) rate at which we thrash it forcing too many stable major numbers downstream; (b) risk inherent in our seeming to leave (or be a party to leaving) unmaintained stable libs downstream.

/be

As far as this bug is concerned: if we're going to provide ABI version numbers, we should try to make them mean what they're supposed to mean. We're not willing to keep the ABI stable, and we don't want the bad PR of frequently ABI number changes. That sounds like WONTFIX to me.

Of course, Ubuntu and everyone else is free to add promises of their own, but it sounds like it supporting that effort would tie us down in ways we can't afford.

(In reply to comment #6)
> All these things happened or will happen in trunk. I don't think any of them
> are likely to be backported to a stable branch.
>
> If we ever do want to backport a breaking change to a stable branch, we could
> just bump the major version number. But that has not happened much in the past
> and I don't expect it to happen much in the future.

I guess I'm not sure we should be in the business of providing a stable branch. It doesn't seem to fit our priorities (speed and language leadership).

I concur completely with comment 15 here: I think we should WONTFIX the request in comment 0, but since it's already possible for Ubuntu to provide libmozjs however they choose, the resolution of the summary would be WORKSFORME?

Comment 5 was arguing based on JS API needing to change rapidly that we cannot afford to be in the business of providing a stable branch, but there are other reasons too. I agree with the conclusion in comment 15. Want to make sure we all agree on the reasoning.

/be

I think a fundamental thing here is that there's a large world between:

[API changes all the time] -- ... -- [ stable API/ABI ]

In particular, in the middle area, there are small things that the SpiderMonkey consuming community would greatly benefit from that don't restrict SpiderMonkey really. See https://bugzilla.mozilla.org/show_bug.cgi?id=618381 for an example.

Another one is build support code for an independent-of-firefox SpiderMonkey release could be upstream, and it could have an independent version.

So, let me lay out some tasks:

* Land some version of 618318 so that embedders can adapt more easily when API changes
* Move xulrunner/installer/mozilla-js.pc.in into js/src/
* Add a version to the libmozjs.so
* Remove the Firefox configure flag --enable-shared-js
* Add a "make dist" target that generates spidermonkey-X.tar.gz to js/src/Makefile.in

(In reply to comment #18)
> * Remove the Firefox configure flag --enable-shared-js

Why would we do this?

(In reply to comment #19)
> (In reply to comment #18)
> > * Remove the Firefox configure flag --enable-shared-js
>
> Why would we do this?

Because it conceptually conflicts with a separate spidermonkey, assuming we keep the same .so and .pc names. And if you were just interested in consuming spidermonkey, why wouldn't you use /usr/lib/libmozjs.so.42 instead of being coupled to Firefox's /usr/lib/xulrunner-1.9.x/libmozjs.so?

(In reply to comment #20)
> Because it conceptually conflicts with a separate spidermonkey, assuming we
> keep the same .so and .pc names. And if you were just interested in consuming
> spidermonkey, why wouldn't you use /usr/lib/libmozjs.so.42 instead of being
> coupled to Firefox's /usr/lib/xulrunner-1.9.x/libmozjs.so?

Nothing prevents you to build spidermonkey from firefox and ship it in /usr/lib. That's what we do in Debian, and it has worked for a quite long time.

(In reply to comment #18)
> * Add a "make dist" target that generates spidermonkey-X.tar.gz to
> js/src/Makefile.in

AFAIK, there is no make dist target to generate source tarballs for other products. And there used to be spidermonkey releases, though it's been a while:
ftp://ftp.mozilla.org/pub/mozilla.org/js/

(In reply to comment #21)

> Nothing prevents you to build spidermonkey from firefox and ship it in
> /usr/lib. That's what we do in Debian, and it has worked for a quite long time.

Sure, though clearly separate tarballs are better. The larger question is what you do about the mozilla-js.pc file; in this case then, xulrunner-dev wouldn't be providing it, right? Or do you have separate .pc files?

(In reply to comment #20)
> Because it conceptually conflicts with a separate spidermonkey, assuming we
> keep the same .so and .pc names. And if you were just interested in consuming
> spidermonkey, why wouldn't you use /usr/lib/libmozjs.so.42 instead of being
> coupled to Firefox's /usr/lib/xulrunner-1.9.x/libmozjs.so?

Note that conceptually, you'd want a --with-system-js, but I'm already not convinced bug 618318 will actually work out, so I'm doubtful about having an up-to-date configure test.

(In reply to comment #23)
> Sure, though clearly separate tarballs are better. The larger question is what
> you do about the mozilla-js.pc file; in this case then, xulrunner-dev wouldn't
> be providing it, right? Or do you have separate .pc files?

$ apt-file show libmozjs-dev | grep pkgconfig
libmozjs-dev: /usr/lib/pkgconfig/mozilla-js.pc
$ apt-file show xulrunner-dev | grep pkgconfig
xulrunner-dev: /usr/lib/pkgconfig/libxul-embedding-unstable.pc
xulrunner-dev: /usr/lib/pkgconfig/libxul-embedding.pc
xulrunner-dev: /usr/lib/pkgconfig/libxul-unstable.pc
xulrunner-dev: /usr/lib/pkgconfig/libxul.pc
xulrunner-dev: /usr/lib/pkgconfig/mozilla-gtkmozembed-embedding.pc
xulrunner-dev: /usr/lib/pkgconfig/mozilla-gtkmozembed.pc
xulrunner-dev: /usr/lib/pkgconfig/mozilla-plugin.pc

(In reply to comment #23)
> (In reply to comment #21)
>
> > Nothing prevents you to build spidermonkey from firefox and ship it in
> > /usr/lib. That's what we do in Debian, and it has worked for a quite long time.
>
> Sure, though clearly separate tarballs are better.

They also complicate things. See the latest security release (3.5.16/3.6.13), for instance: spidermonkey adds new symbols that xpconnect (in firefox) uses.

(In reply to comment #25)
>
> $ apt-file show libmozjs-dev | grep pkgconfig
> libmozjs-dev: /usr/lib/pkgconfig/mozilla-js.pc

Ok. Does your libxul.so (xulrunner package) depend on libmozjs.so (libmozjs package), or are they on potentially separate schedules?

(In reply to comment #27)
> Ok. Does your libxul.so (xulrunner package) depend on libmozjs.so (libmozjs
> package), or are they on potentially separate schedules?

Package: xulrunner-1.9.2
Depends: (...) libmozjs3d (= 1.9.2.13-1) (...)

(I even force a strict dependency, version wise, though not strictly necessary)

(In reply to comment #28)
> Package: xulrunner-1.9.2
> Depends: (...) libmozjs3d (= 1.9.2.13-1) (...)
>
> (I even force a strict dependency, version wise, though not strictly necessary)

(Both packages come from the same source (iceweasel) package)

Bug 618318 is a little bit of a different bug. It doesn't make us 'grow up and become a ral library', but it goes about halfway there in a way which is useful for quite a number of users. Specifically, it:

- packages spidermonkey separately to FF and xulrunner.
- adds a major version number to the .so file, which is updated on API (and ABI as much as we can) changes
- allows mozilla to keep making incompatible changes, with a new .so number
- allows distributions to backport changes if they wish (in which case they would update the minor version number).
- only adds a tiny burden to mozilla (remember to increase the number of API changes, and aspirationally ABI changes).

It doesn't do everything distros would like (too high a burden on us), but it makes the world nicer for users.

I propose we WONTFIX this bug, and move our attention to bug 618318.

Wrong bug#?

(In reply to comment #31)
> Wrong bug#?

My bad, bug 618381.