Could not create git repository in personal namespace

Easily reproduced:

  $ git clone git+ssh://<email address hidden>/charm-telegraf
  $ cd charm-telegraf
  $ git push git+ssh://<email address hidden>/~jsing/charm-telegraf master
  fatal: remote error: Could not create repository: [Errno 13] Permission denied

Note that the error message in Joel's output is not the same as that in Bryce's. (But Thiago's exception-raising tweaks may have fixed Bryce's issue, or at least changed it.)

2020-10-01 07:59:11+0000 [-] [request-id=c4a95379-a57c-4238-b2fb-08d1a93def9d] [PackBackendProtocol] Request received: 'turnip-create-repo 58934', params=OrderedDict([('turnip-authenticated-user', 'jsing'), ('version', '0'), ('clone_from', '46512'), ('turnip-authenticated-uid', '5240003'), ('turnip-request-id', 'c4a95379-a57c-4238-b2fb-08d1a93def9d')])
2020-10-01 07:59:11+0000 [-] [request-id=c4a95379-a57c-4238-b2fb-08d1a93def9d] [PackBackendProtocol] Creating repository: 58934
2020-10-01 07:59:11+0000 [-] [request-id=c4a95379-a57c-4238-b2fb-08d1a93def9d] [PackBackendProtocol] Creating repository /srv/turnip/data/repos/58934, clone of /srv/turnip/data/repos/46512
2020-10-01 07:59:12+0000 [-] [request-id=c4a95379-a57c-4238-b2fb-08d1a93def9d] [PackBackendProtocol] Aborting on Launchpad repo /srv/turnip/data/repos/58934 creation: [Errno 13] Permission denied
2020-10-01 07:59:13+0000 [-] [request-id=c4a95379-a57c-4238-b2fb-08d1a93def9d] [PackBackendProtocol] Deleting local repo creation attempt /srv/turnip/data/repos/58934.
2020-10-01 07:59:13+0000 [-] [request-id=c4a95379-a57c-4238-b2fb-08d1a93def9d] [PackBackendProtocol] Dying: Could not create repository: [Errno 13] Permission denied
2020-10-01 07:59:13+0000 [-] [request-id=c4a95379-a57c-4238-b2fb-08d1a93def9d] [PackBackendProtocol] Request received: 'git-receive-pack 58934', params=OrderedDict([('turnip-authenticated-user', 'jsing'), ('version', '0'), ('turnip-authenticated-uid', '5240003'), ('turnip-request-id', 'c4a95379-a57c-4238-b2fb-08d1a93def9d')])
2020-10-01 07:59:13+0000 [-] [request-id=c4a95379-a57c-4238-b2fb-08d1a93def9d] [PackBackendProtocol] Connection closed.

Indeed, later in the day yesterday I tried pushing again and it worked (and then I pushed a new branch for a different package, with no troubles.) I briefly looked at Thiago's git commits but it wasn't obvious to me what the underlying issue had been or why it started working, but the fast attention given to it is much appreciated. Whatever the issue was for me, it seems not reproducing now and I presume Thiago's changes fixed it.

Note that my 'bryce' remote is defined as pointing to inside the ubuntu project space, i.e.:

[remote "bryce"]
        url = https://git.launchpad.net/~bryce/ubuntu/+source/php-parser
        fetch = +refs/heads/*:refs/remotes/bryce/*
        fetch = +refs/tags/*:refs/tags/bryce/*
        pushurl = ssh://<email address hidden>/~bryce/ubuntu/+source/php-parser
        tagOpt = --no-tags

I suspect there's some subtle differences between git branches in ~user/ vs ~user/ubuntu/+source/ but don't know if that would affect things.

FTR, I did not make any changes to ssh keys, gpg keys, VPN or other launchpad account settings yesterday. I did re-sign in to SSO once or twice yesterday, but couldn't say if that was related to the problem or not.

To be fair, my changes were more about adding more information than fixing the problem. Since it cannot be reproduced in QA nor locally, it seems to be something specific to the production environment.

It happened again to me few minutes ago, after adding more logs:

```
2020-10-01 18:00:23+0000 [-] [request-id=8361bb32-1afd-488b-83ac-a648a772d641] [PackBackendProtocol] Creating repository: 58959
2020-10-01 18:00:23+0000 [-] [request-id=8361bb32-1afd-488b-83ac-a648a772d641] [PackBackendProtocol] Creating repository /srv/turnip/data/repos/58959, clone of /srv/turnip/data/repos/17181
2020-10-01 18:00:23+0000 [-] [request-id=8361bb32-1afd-488b-83ac-a648a772d641] [PackBackendProtocol] Running init_repository(/srv/turnip/data/repos/58959, True)
2020-10-01 18:00:24+0000 [-] [request-id=8361bb32-1afd-488b-83ac-a648a772d641] [PackBackendProtocol] Running set_repository_creating(/srv/turnip/data/repos/58959, True)
2020-10-01 18:00:24+0000 [-] [request-id=8361bb32-1afd-488b-83ac-a648a772d641] [PackBackendProtocol] Running init_repository for subordinate /srv/turnip/data/repos/58959/turnip-subordinate
2020-10-01 18:00:25+0000 [-] [request-id=8361bb32-1afd-488b-83ac-a648a772d641] [PackBackendProtocol] Aborting on Launchpad repo /srv/turnip/data/repos/58959 creation: [Errno 13] Permission
 denied.
          File "/srv/turnip/turnip-pack-backend/payloads/4e5e9da6c5eb5679ae65465b02635f5de3594a58/turnip/pack/git.py", line 545, in _createRepo
            log=self.log)
          File "/srv/turnip/turnip-pack-backend/payloads/4e5e9da6c5eb5679ae65465b02635f5de3594a58/turnip/api/store.py", line 296, in init_repo
            packable_refs.update(import_into_subordinate(sub_path, clone_from))
          File "/srv/turnip/turnip-pack-backend/payloads/4e5e9da6c5eb5679ae65465b02635f5de3594a58/turnip/api/store.py", line 252, in import_into_subordinate
            os.link(from_path, sub_path)
```

It's curious that right before this failure, the same git push worked.

It's probably something in the environment itself. I'll see if a restart in our NFS service will make the issue go away. I'll just wait for a "quieter" moment to do that, to reduce the impact for the other users.

William pointed out that this was a recurrence of https://bugs.launchpad.net/turnip/+bug/1853174, and we had this documented in https://wiki.canonical.com/InformationInfrastructure/WebOps/LP/LaunchpadGitDeployment#nfs-ganesha_vs_apparmor but had forgotten about this when we had to reboot one of the physical hosts recently. Sorry about that! We've reapplied the workaround, and it should work a bit better now, although the original bug remains open.