I also did some testing with snapshots created via the documented process. If you shut down etcd manually on the unit and touch the member/snap/db file as mentioned in the above ticket, when etcd is restarted it does allow the process to proceed.
Also - when the snapshot is restored, the cluster memberships are restored with it. This causes another etcd crash if restoring to a different environment (or a rebuild of the existing one) where the etcd units have changed.
Testing with snapshots from a different environment (with the above workarounds) I was able to get the data restored and verify the k8s data was present.
I did not run the action with keys-version=v3. I was unaware of that option (it's not in the docs FYI)
I did some investigation yesterday and I found a few things.
The missing db file seems to be a bug in pre 3.1.8 etcd itself (we have 3.0.17 in the affected environment)
https:/ /github. com/etcd- io/etcd/ issues/ 8331
I also did some testing with snapshots created via the documented process. If you shut down etcd manually on the unit and touch the member/snap/db file as mentioned in the above ticket, when etcd is restarted it does allow the process to proceed.
Also - when the snapshot is restored, the cluster memberships are restored with it. This causes another etcd crash if restoring to a different environment (or a rebuild of the existing one) where the etcd units have changed.
Testing with snapshots from a different environment (with the above workarounds) I was able to get the data restored and verify the k8s data was present.