Evergreen

Vandelay record queuing can fail if spool directory is /tmp

Bug #1855199 reported by Galen Charlton on 2019-12-04

This bug affects 7 people

Affects		Status	Importance	Assigned to	Milestone
	Evergreen	Confirmed	Medium	Unassigned

Bug Description

On recent Debian and Ubuntu distributions, leaving the opensrf.xml setting open-ils.vandelay/app_settings/databases/importer set to its default value of /tmp can cause uploading MARC records into a queue to fail.

This is because (at least) Debian Stretch and later and Ubuntu 17.04 and later [1], the systemd PrivateTmp option is enabled by default for the apache2 service. When this option is in effect, anything that an Apache process writes to /tmp is not visible to other processes, including open-ils.vandelay.

This can be worked around by setting the scratch directory to something like /openils/var/tmp. Unless there happens to be a systemd-approved way of securely saying that the contents of the Apache PrivateTmp should be visible to OpenSRF processes, we're either going to have to change the default scratch path in opensrf.xml or update the installation instructions to recommend turning off PrivateTmp.

[1] https://muras.eu/2017/12/06/apache-ubuntu-systemd-privatetmp/

This affects any Evergreen version that's running on Debian 9+ or Ubuntu 17.04+

Tags:

Revision history for this message

Galen Charlton (gmc) wrote on 2019-12-04:

Upon a bit of searching, it looks like using JoinsNamespaceOf would be one of way of managing this in a systemd-approved manner (that is, of course, in scenarios where you're also using systemd unit files to manage the OpenSRF services).

Changed in evergreen:
importance:	Undecided → Medium

Revision history for this message

Bill Erickson (berick) wrote on 2019-12-05:

Thanks, Galen. Had an issue on 18.04 recently but had not yet investigated.

Changed in evergreen:
status:	New → Confirmed

Tiffany Little (tslittle) on 2021-04-22

tags:

added: vandelay

Revision history for this message

Dan Wells (dbw2) wrote on 2021-09-27:

Ran into this today. It is also very likely the problem for a few of the folks on this thread, though it was only ever diagnosed in a roundabout way (i.e. never pointed back the the systemd PrivateTmp feature): https://<email address hidden>/msg00302.html

Seems like moving the default to something like /openils/var/tmp/ as suggested (or even just /openils/tmp/) is a simple and sane path forward.

Revision history for this message

Jason Stephenson (jstephenson) wrote on 2021-09-29:

I'm in favor of moving the default in the sample config to /openils/var/tmp and making sure the directory exists during installation.

I think this is a more portable solution than adding systemd units. We may some day get Evergreen working on other UNIX-like operating systems.

Elaine Hardy (ehardy) on 2021-10-15

tags:

added: acq-loadmarc cat-importexport
removed: vandelay

Revision history for this message

Jason Boyer (jboyer) wrote on 2022-10-05:

Here's a branch that fixes the problem for systems regardless of systemd use:
https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/jboyer/lp1855199_nice_updo / working/user/jboyer/lp1855199_nice_updo

It uses /openils/var/data/vandelay to follow the example set by offline uploads. Also, I think the use of a "tmp" dir implies things only temporarily needed on the local machine; I'm not a fan of using that over a (potential) NFS share.

tags:

added: pullrequest

Bill Erickson (berick) on 2023-03-01

Changed in evergreen:
assignee:	nobody → Bill Erickson (berick)

Revision history for this message

Bill Erickson (berick) wrote on 2023-03-01:

Jason, looks like the new directory is going to: /openils/share/vandelay (There's an offline directory there as well). Update the sample config?

Changed in evergreen:
assignee:	Bill Erickson (berick) → nobody

Revision history for this message

Jason Boyer (jboyer) wrote on 2023-03-01:

I've apparently been putting offline under var/data myself in the past and not realized it when putting this together. The confusion is coming from $datadir being set to @localstatedir@/data in Makefile.am, but once Makefile comes out the other end it's changed to $datarootdir. So, I expect that it's a case of not doing things the expected way with autotools.

FHS-wise, I'd think that the (actual) data dir under $PREFIX/var is the way to go. (Things in share/ are supposed to be read-only and I continue to harbor the delightful fiction that someday we'll be able to build deb packages that follow the rules enough to someday be included in main)

But that's a bug for another day. :-/

tags:

removed: pullrequest