nagios failure "check-graylog-health" fails with Indexer failures 69000

Running against the latest/stable graylog charm, and using the graylog 3/stable channel for the snap SQA had all the nagios units fail the "check-graylog-health" check because of "Indexer failures: 69000"

we run elasticsearch with the settings `auto-create-index: .watches,.triggered_watches,.watcher-history-*`

elastic is on an 12G mem and 2 CPU system with 50G disk, and graylog is on a 8G mem, 2 CPU system with a 40G disk.

Nothing stands out in the elastic or graylog logs as to what the problem may be further than this.
outputs from nagios are:

- check: check-graylog-health
  id: '124'
  results:
    check-output: 'CRITICAL: Indexer failures: 69000

      OK: Indexer cluster health: green; Journal uncommitted messages: 3; Outstanding
      notifications: 0'
    return-code: 0
  status: completed
  timing:
    completed: 2022-12-07 07:59:37 +0000 UTC
    enqueued: 2022-12-07 07:59:36 +0000 UTC
    started: 2022-12-07 07:59:37 +0000 UTC
  unit: nrpe/7
- check: check-graylog-health
  id: '126'
  results:
    check-output: 'CRITICAL: Indexer failures: 69000

      OK: Indexer cluster health: green; Journal uncommitted messages: 0; Outstanding
      notifications: 0'
    return-code: 0
  status: completed
  timing:
    completed: 2022-12-07 07:59:37 +0000 UTC
    enqueued: 2022-12-07 07:59:36 +0000 UTC
    started: 2022-12-07 07:59:37 +0000 UTC
  unit: nrpe/6
- check: check-graylog-health
  id: '121'
  results:
    check-output: 'CRITICAL: Indexer failures: 69000

      OK: Indexer cluster health: green; Journal uncommitted messages: 103; Outstanding
      notifications: 0'
    return-code: 0
  status: completed
  timing:
    completed: 2022-12-07 07:59:37 +0000 UTC
    enqueued: 2022-12-07 07:59:36 +0000 UTC
    started: 2022-12-07 07:59:37 +0000 UTC
  unit: nrpe/1

testrun can be found at:
https://solutions.qa.canonical.com/v2/testruns/6e60d071-7912-492a-8784-3c53d7d532ce/
with bundle at:
https://oil-jenkins.canonical.com/artifacts/6e60d071-7912-492a-8784-3c53d7d532ce/generated/generated/lma-openstack/bundle.yaml
and crashdump at:
https://oil-jenkins.canonical.com/artifacts/6e60d071-7912-492a-8784-3c53d7d532ce/generated/generated/lma-maas/juju-crashdump-lma-maas-2022-12-07-08.01.03.tar.gz