GCE credential 403 error spams logs

Thanks for raising this Ian. We'll look to address this as soon as possible.

I am not convinced that the re-try should happen on 'juju status'... What if a user is running 'watch juju status'? If nothing changes on cloud side, you'd still be spamming the logs.

I think a cleaner way is to have a command to re-try current credential...

Currently, in order to flip the credential back to 'valid' on juju side, we need to trigger a re-upload of local credential which is done either via run 'update-credential' or 'add-model --credential'.

PR against develop: https://github.com/juju/juju/pull/10200

I've attempted several things to force the GCE credentials into an error state. If an account is deleted, Google returns an error, but Juju continually retries:

machine-7: 15:47:23 INFO juju.worker.machiner "machine-7" started
machine-7: 15:47:23 ERROR juju.worker.dependency "machiner" manifold worker returned unexpected error: cannot update observed network config: cannot get network interfaces of "juju-eaa547-7": Get https://www.googleapis.com/compute/v1/projects/juju-qa-checking-broken-creds/aggregated/instances?alt=json&filter=name+eq+juju-eaa547-.%2A: oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error": "invalid_grant",
  "error_description": "Not a valid email or user ID."
}

That's interesting... I am pretty sure that 400 Bad Request is catered for in the code to invalidate a credential.

Thank you for taking a look!

If a service account with insufficient permissions as a Juju credential, the credential is marked as invalid quickly.

ERROR closing port(s) [17070/tcp]: googleapi: Error 403: Required 'compute.firewalls.delete' permission for 'projects/juju-qa-checking-broken-creds/global/firewalls/juju-845bc02a-4fd0-4cd8-82c4-33262c385ced'
More details:
Reason: forbidden, Message: Required 'compute.firewalls.delete' permission for 'projects/juju-qa-checking-broken-creds/global/firewalls/juju-845bc02a-4fd0-4cd8-82c4-33262c385ced'
Reason: forbidden, Message: Required 'compute.networks.updatePolicy' permission for 'projects/juju-qa-checking-broken-creds/global/networks/default'

I've confirmed that the PR referenced above prevents the requests in a loop.

Attempting to bootstrap with a valid credential when the Google Compute API is disabled halts immediately.

Note that Google may have updated their internal systems to prevent a similar situation. It is now impossible to revoke access to the API if resources exist that were created via the API.