Cilium leader fails hook "send-remote-write-relation-joined" due to missing .kube/config

Bug #2025162 reported by Bas de Bruijne
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cilium Charm
Fix Released
High
Mateo Florido

Bug Description

(related to LP: #2025157)

When deploying cilium related to cos-prometheus the installation fails with hook `failed: "send-remote-write-relation-joined"`:

========
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/lightkube/config/kubeconfig.py", line 175, in from_service_account
    token = account_dir.joinpath("token").read_text()
  File "/usr/lib/python3.10/pathlib.py", line 1134, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/var/run/secrets/kubernetes.io/serviceaccount/token'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/lightkube/config/kubeconfig.py", line 210, in from_env
    return KubeConfig.from_service_account(service_account=service_account)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/lightkube/config/kubeconfig.py", line 178, in from_service_account
    raise exceptions.ConfigError(str(e))
lightkube.core.exceptions.ConfigError: [Errno 2] No such file or directory: '/var/run/secrets/kubernetes.io/serviceaccount/token'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cilium-0/charm/./src/charm.py", line 336, in <module>
    main(CiliumCharm)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/ops/framework.py", line 354, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/ops/framework.py", line 830, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/ops/framework.py", line 919, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cilium-0/charm/lib/charms/prometheus_k8s/v0/prometheus_remote_write.py", line 673, in _handle_endpoints_changed
    self.on.endpoints_changed.emit(relation_id=event.relation.id)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/ops/framework.py", line 354, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/ops/framework.py", line 830, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/ops/framework.py", line 919, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cilium-0/charm/./src/charm.py", line 266, in _on_remote_write_changed
    self._deploy_grafana_agent(self.remote_write_consumer.endpoints)
  File "/var/lib/juju/agents/unit-cilium-0/charm/./src/charm.py", line 172, in _deploy_grafana_agent
    self._client.apply(obj)
  File "/usr/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/var/lib/juju/agents/unit-cilium-0/charm/./src/charm.py", line 101, in _client
    client = Client(field_manager=f"{self.model.app.name}")
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/lightkube/core/client.py", line 43, in __init__
    self._client = GenericSyncClient(config, namespace=namespace, timeout=timeout, lazy=lazy,
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/lightkube/core/generic_client.py", line 82, in __init__
    config = KubeConfig.from_env().get()
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/lightkube/config/kubeconfig.py", line 212, in from_env
    return KubeConfig.from_file(os.environ.get('KUBECONFIG', default_config))
  File "/var/lib/juju/agents/unit-cilium-0/charm/venv/lightkube/config/kubeconfig.py", line 142, in from_file
    raise exceptions.ConfigError(f"Configuration file {fname} not found")
lightkube.core.exceptions.ConfigError: Configuration file ~/.kube/config not found
========

The error message is very descriptive, but the preferred behaviour would be to retry the hook later when the kube/config is synchronized.

tags: added: cdo-qa foundations-engine
George Kraft (cynerva)
Changed in charm-cilium:
importance: Undecided → High
status: New → Triaged
Changed in charm-cilium:
assignee: nobody → Mateo Florido (mateoflorido)
status: Triaged → In Progress
Changed in charm-cilium:
status: In Progress → Fix Committed
Changed in charm-cilium:
milestone: none → 1.28
Adam Dyess (addyess)
Changed in charm-cilium:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.