DB CA link did not reconnect (is this same as #153 ????)

I have two questions for you.

1) What version of EPICS runs on scl-hprf-ioc05?

2) Is this is a side effect of Kay’s IOCs hanging up during the shutdown part of a soft reboot? If that was occurring then we certainly do understand why clients would not reconnect?

If *all* of the following are true the client might not reconnect on its own.

1) client side is running R3.14.4 (possibly R3.14.5) or earlier

2) the server was unavailable for longer than 10 – 30 min (depending on EPICS version and IP kernel version)

3) the server isn’t configured so the client sees the server’s beacon, or the router does not forward the server’s beacon to the client (the client sees no changes in the frequency of server beacons)

Also, looking in mantis there was a bug fixed in the R3.14.2 – R3.14.4 (inclusive) CA repeater that might cause a problem like you have described.

So if your client side (scl-hprf-ioc05) was running R3.14.6 or later and Kay’s IOC was not hung when this was supposed to reconnect then it sounds like I will need to investigate further. Let me know.

From Pam,

All of the IOCs involved are running 3.14.7.

No – I don’t think it’s the same IOC that Kay was having the hangup problem on. I don’t know how long his reboot took.

From Pam,

Also, this problem persisted overnight until I came in the following morning - indeed, until the IOC was rebooted. It seems likely in this case that something was messed up so it wasn't making new connections (or remaking old ones.)

It was fixed by a reboot of the IOC with the linking records.

I haven't seen a problem quite like this one before, so maybe we should just hope it doesn't happen again.

However, I have seen, with 3.14.7 soft IOCs, records taking quite long times to reconnect - 5 minutes and more. Also I've seen that after restarting the archiver.

From Kay,

I doubt it's related to the LLRF reboot:
This is about rebooting about 40 IOCs in order to load new software in the afternoon by writing to some PV which runs 'reboot()'. There are terminal windows open to most of the IOCs, so I see that one IOC doesn't want to die & reboot without a Ctrl^X. All in all this happens quickly; all LLRF IOCs are back up within 5 minutes.

This is worrisome. I went ahead and created Mantis entry 206 so that we can track this issue.

Of course the IOC was rebooted and so it's too late now, but next we will need to have a look at the stack trace for that timer queue. This is accomplished on vxWorks by typing "tt <task id>". One possibility might be that the timer queue thread is stuck in one of its callbacks and this is hanging up the reconnect mechanism.

> However, I have seen, with 3.14.7 soft IOCs, records taking quite
> long times to reconnect - 5 minutes and more. Also I've seen that
> after restarting the archiver.

If the client sees the servers beacon it should reconnect within a few seconds (maxing out at 5 seconds as I recall). With old EPICS it was possible for it to not reconnect if the servers beacon was not seen and the IOC was inaccessible for a long time. With recent R3.14, if it does not see the servers beacon, it should always reconnect after a maximum of about 5 minutes.

One possibility is that the IP kernel is not disconnecting the TCP circuit when the server (IOC) was rebooted. When CA notices that there is no activity over the circuit and no beacon for EPICS_CA_CONN_TMO it sends an "are you there" message over the TCP circuit. Given that the "are you there" message is being attempted to be delivered by the IP kernel then the IP kernel should receive a "no circuit at that port" ICMP response immediately when the IOC reboots, and that should cause it to disconnect the circuit, and that should cause the client library to build a new circuit.

What version of vxWorks are the IOCs running? Do we have a mixture of vxWorks 6.0 and other vxWorks versions?

edited on: 2005-06-15 11:15

From Kay,

Maybe related but too late now:
There was work being done on the event link hardware.
As a result, a sequence of LLRF IOCs didn't
receive time-of-day info from the SNS event system
and their "bootup" time showed 1-1-1990.

Their current time was OK because the now famous
General Time aka Colonel Time alias Chairman Time
fell back to NTP and the LLRF IOCs worked OK
until I needed to reboot for software updates.

Maybe that HPRF IOC wasn't so lucky and the timer
queue happened to run into bad time info while
calculating delays.

This is just to hint that there's so much going on
in parallel that it's hard to debug.

edited on: 2005-06-15 11:19

Yea, we are vulnerable if there are absolute time source discontinuities. I don’t know how to avoid this with timer queues. We have to work in absolute time because we don’t know how much time goes by when users are in their callbacks.

I do know that NTP has many years ago solved the discontinuities problem. They use a software updated PLL to gracefully slew from one time source to another.

These IOCs are all running vxWorks "SNS07" or Wind version 2.6

The appears to have been an issue with the time distribution, which hopefully cannot reccur with the integration of generalTime into the IOC.

R3.14.10 released.