Retire iocLogserver

My logstash configuration that replaces the iocLogServer program looks like this:

input {
    tcp {
        host => "${EPICS_IOC_LOG_INET:localhost}"
        port => "${EPICS_IOC_LOG_PORT:7004}"
        id => "iocLog-${EPICS_IOC_LOG_PORT:7004}"
        codec => "line"
    }
}

output {
    file {
        path => "/path/to/logs/ioc-%{+YYYY-MM-dd}.json"
    }
}

That output configuration creates a new file every day, but sites may send the messages straight off to their log database instead.

I'm wondering whether we might want to adjust the errlog code to support the multiline input codec by ensuring that there is a space after any '\n' that appears before the end of a message. I haven't found many multi-line messages which would need that so maybe it should be done by hand instead.

The epicsStackTrace.c code will not work well with the single-line codec but could be easily adjusted to work with the multiline one; there may be other examples that I'm not aware of. Here's the codec configuration I suggest we aim to support:

        codec => multiline {
            pattern => "^\s"
            what => "previous"
        }

> I'm wondering whether we might want to adjust the errlog code to support the multiline input codec

Or maybe escaping is a better idea? We should probably be doing this anyway for non-printable characters.

Also, keeping to a single line gives (practical) compatibility with any tool which understands syslog

https://tools.ietf.org/html/rfc5424

eg. syslog-ng wants one line per message over tcp

https://github.com/balabit/syslog-ng/issues/1747

The errlogClient doesn't maintain message boundaries, it just collects data in a buffer and passes it to send() when the buffer gets full or someone calls flush. If you look at that epicsStackTrace.c code you'll see that it calls errlogPrintf() in a loop, once for each stack item, with newline. This one TCP message from logClient.c can contain more or less than one logical errlog item. Compatibility with syslog would thus seem to require rather more changes than the alternatives.

My 2 centimes:

Removing a widely used binary that's been around for decades as part of a bugfix release, without warning or prior announcement, is rude and completely out of question.

For now (i.e. 7.0.2), I would support printing a deprecation warning on the console and into the log when starting iocLogserver.
A minor release (7.1) could have iocLogServer as a script that calls a plug-in-compatible replacement.
Complete removal will have to wait for a major release.

Such a change (iocLogserver -> logstash) requires *major* changes for packaged versions of Base and any productional systems.

I agree that we can't remove iocLogserver as part of a bugfix release. I would suggest the following approach:

1. Test and document new solutions like Logstash.
2. Decide if and when to retire iocLogserver. The decision should be made soon, the removal date far enough in the future to give users time to move on to new tools.
3. Add deprecation warning (pointing users to the instructions for setting up new solutions), announce on tech-talk and at an EPICS meeting.
4. Retire iocLogserver as planned.

I started working on (1): https://docs.google.com/document/d/1cgMWBA_fF3vrxru5N-NOU1_kNTr-sKLpKsUPO0Dxnz0/edit?usp=sharing Please contribute. Where would be a good place to put this information once it is complete? The new EPICS web site?

Core Group review at ESS: Retiring is the direction we want to go, but actual removal of the server will need to wait for a major release. This document should become a HowTo page on the new epics-controls website.

AI on AJ: Add a note to this effect to the Release Notes (would be nice to be able to link to the howTo page).

While merging the fix for #1786858 I also added the agreed-on comment about retirement.

We didn't agree on when we would actually retire the iocLogServer, I assume it should be in 7.0.