Cannot share a namespaces created with 'ip netns' between apps in a devmode SNAP

Bug #1611444 reported by Dave Garrod
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Snappy
Invalid
High
Zygmunt Krynicki
snap-confine
Fix Released
Critical
Zygmunt Krynicki
snap-confine (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

snap-confine uses linux namespaces (specifically the mount namespace) to give each started snap application process an isolated an unique view of the filesystem. This prevented applications using namespaces through bind mounted files, e.g. using the "ip netns" command as any changes to the namespace would be "locked" in the unique mount namespace of each application process.

Now snap-confine is re-designed to put all applications belonging to a given snap in the same mount namespace. The first started application creates and persists the mount namespace (in a way similar to running the command: unshare -m /path/to/file) and all other processes for all apps in the same snap just join that populated namespace.

For more information about the execution environment, please see this article http://www.zygoon.pl/2016/08/snap-execution-environment.html

[Test Case]

The test case can be found here:

https://github.com/snapcore/snap-confine/blob/master/spread-tests/main/mount-ns-sharing/task.yaml

The test case is ran automatically for each pull request and for each final release. It can be reproduced manually by executing the shell commands listed in the prepare/execute/restore phases manually.
The commands there assume that snapd and snap-confine are installed.
No other additional setup is necessary.

Note that this feature affects every application in every snap.

[Regression Potential]

 * Regression potential is moderate. This change is large and intrusive and has managed to uncover bugs in the kernel implementation of apparmor (e.g. https://bugs.launchpad.net/apparmor/+bug/1624497)

The feature was tested extensively by the upstream developers but still a potential for unexpected breakage is significant.

[Other Info]

* This bug is a part of a major SRU that brings snap-confine in Ubuntu 16.04 in line with the current upstream release 1.0.41.

* snap-confine is technically an integral part of snapd which has an SRU exception and is allowed to introduce new features and take advantage of accelerated procedure. For more information see https://wiki.ubuntu.com/SnapdUpdates

== # Pre-SRU bug description follows # ==

Please see:

https://<email address hidden>/msg00542.html

for additional details. It was requested that I move that discussion to this bug report.

In summary it appears that multions "apps" in a SNAP cannot share the same NETNS namespace. If one app create a namespace the other apps in SNAP cannot use it. They get assorted errors like:

RTNETLINK answers: Invalid argument

Please see the details in the mail archive posting.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

What version of snap-confine/ubuntu-core-launcher are you using? I ask because I noticed a bug in snap-confine in xenial-proposed where CLONE_NEWUSER got EPERM but with ubuntu-core-launcher from xenial release it worked. Curious if these are at all related.

Please provide output from:
$ apt-cache policy ubuntu-core-launcher snap-confine

Revision history for this message
Dave Garrod (dgarrod) wrote :

Re:

> Please provide output from:
> $ apt-cache policy ubuntu-core-launcher snap-confine

Here is the requested output on system wghere we install the SNAP. It appears there is no such thing as "snap-confine"

netop@switch:~$ apt-cache policy ubuntu-core-launcher snap-confine
ubuntu-core-launcher:
  Installed: 1.0.27.1
  Candidate: 1.0.27.1
  Version table:
 *** 1.0.27.1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.27 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
N: Unable to locate package snap-confine

Exact same think on the system where we build the SNAP

bldecos-4:/builds/dgarrod/ops-build-snx% apt-cache policy ubuntu-core-launcher snap-confine
ubuntu-core-launcher:
  Installed: 1.0.27.1
  Candidate: 1.0.27.1
  Version table:
 *** 1.0.27.1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.27 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
N: Unable to locate package snap-confine

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Thank you for responding-- this is clearly not related to the CLONE_NEWUSER issue then.

Revision history for this message
Dave Garrod (dgarrod) wrote :

Please could someone comment on this issue or at least point me in a direction I can look for a solution. This problem is really holding up our ability to SNAPify OpenSwitch.

Thanks,

Dave

Revision history for this message
Stéphane Graber (stgraber) wrote :

The launcher creates a new mount namespace per snap, which is occasionally a problem as I mentioned in the LXD bug here https://bugs.launchpad.net/snappy/+bug/1606510.

However it does not create a new network namespace per app, otherwise we'd be running into all sorts of extra troubles.

It would be useful to know exactly what your app is doing at that time to figure out why it's being denied by the kernel. I'm assuming you're testing all this in devmode right now and that seccomp and apparmor confinement shouldn't be in the way.

Does it try to move itself into the network namespace (through setns) or just asks netlink to move an interface into a namespace? If the latter, some trace showing where it grabbed the netns fd from would be useful, just to confirm that it did manage to open it and that it sent something valid looking over netlink.

Revision history for this message
Dave Garrod (dgarrod) wrote : RE: [Bug 1611444] Re: Cannot share a namespace between apps in a SNAP

Thank you for your reply. The questions you asked should be answered by my prior post to the listserv at:

https://<email address hidden>/msg00546.html

Specifically commenting on your questions:

Re:

> The launcher creates a new mount namespace per snap, which is occasionally a problem as I mentioned in the LXD bug here
> https://bugs.launchpad.net/snappy/+bug/1606510.
>
> However it does not create a new network namespace per app, otherwise we'd be running into all sorts of extra troubles.

That's exactly how I expected it to work and thus why I don’t understand why it doesn't work for me.

Re:

> It would be useful to know exactly what your app is doing at that time to figure out why it's being denied by the kernel. I'm assuming you're testing
> all this in devmode right now and that seccomp and apparmor confinement shouldn't be in the way.

Indeed I'm currently running the SNAP in --devmode so seccomp and apparmor shouldn't be getting in the way.

Re:

> Does it try to move itself into the network namespace (through setns) or just asks netlink to move an interface into a namespace? If the latter,
> some trace showing where it grabbed the netns fd from would be useful, just to confirm that it did manage to open it and that it sent something
> valid looking over netlink.

The post I referred to above should answer these questions. It shows an strace of the failing command.

In summary I create two "bash" windows in the SNAP. In the first one I create the namespace:

ip netns add xxx

In the second one I try and use it the namespace:

root@Snappy-Appliance:/home/netop# ip netns list
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument

As you can see it fails. The detailed "strace" is in the post referred to above.

Dave

________________________________

DISCLAIMER:
This e-mail and any attachments to it may contain confidential and proprietary material and is solely for the use of the intended recipient. Any review, use, disclosure, distribution or copying of this transmittal is prohibited except by or on behalf of the intended recipient. If you have received this transmittal in error, please notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed.

Revision history for this message
Stéphane Graber (stgraber) wrote : Re: Cannot share a namespace between apps in a SNAP

Oh, so it may be a mount namespace problem after all.

iproute uses bind-mounts of the netns fd onto /run/netns/<somename> (that's what shows up as nsfs).

Because every app launched by snapd runs in a different mount namespace, you won't be able to see those from other apps, so as far as the second app is concerned, all it sees in /run/netns is an empty file, the mount itself is in the other app's namespace.

There is currently no support for sharing the mount namespace between multiple apps within a snap (that's what I mentioned in that bug report I pointed you to earlier). Which would be the easiest solution for you to fix this.

There are certainly ways for you to workaround this limitation but they would either be very hackish (such as doing an rbind from /proc/<other app pid>/root/run/netns onto /run/netns in a wrapper) or require substantial rework of your workload.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I can confirm this. Here is a test snap: https://code.launchpad.net/~jdstrand/+git/snap-netns-test

$ snapcraft
...
$ sudo snap install --devmode snap-netns-test_1_all.snap

See that everything works as expected:
$ sudo /snap/bin/snap-netns-test.sh
bash-4.3# ip netns add testnet
bash-4.3# ls /run/netns/
testnet
bash-4.3# ip netns exec testnet ls /run/netns
testnet
bash-4.3# ip netns del testnet
bash-4.3# ip netns del testnet
bash-4.3# ls /run/netns/

All of the above shows it works fine under a single shell invocation. Now try between two app invocations from within the same snap:

In one terminal:
$ sudo /snap/bin/snap-netns-test.sh
bash-4.3# ip netns add testnet
bash-4.3# ip netns exec testnet ls /run/netns
testnet

In another terminal:
$ sudo /snap/bin/snap-netns-test.sh2
bash-4.3# ip netns exec testnet ls /run/netns
RTNETLINK answers: Invalid argument
setting the network namespace "testnet" failed: Invalid argument

Now strace from within terminal 2:
bash-4.3# strace -f -vv -o $SNAP_DATA/trace ip netns exec testnet ls /run/netns
RTNETLINK answers: Invalid argument
setting the network namespace "testnet" failed: Invalid argument
...
1150 open("/var/run/netns/testnet", O_RDONLY|O_CLOEXEC) = 5
1150 setns(5, CLONE_NEWNET) = -1 EINVAL (Invalid argument)
1150 write(2, "setting the network namespace \"t"..., 65) = 65
1150 close(5) = 0
1150 exit_group(1) = ?
1150 +++ exited with 1 +++
...

Full strace attached.

Changed in snappy:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I also tried this with 'confinement: strict' (adding a bunch of policy to make setting up network namespaces work and for strace) and have an equivalent strace with no denials.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I also confirmed myself that downgrading to ubuntu-core-launcher 1.0.27.1 yields the same error.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

To expand slightly on what Stephane was saying, you can see the problem easily from the snap I provided, in terminal 1 (ie, the one that set the network namespace):

bash-4.3# cat /var/run/netns/testnet
cat: /var/run/netns/testnet: Invalid argument
(shows this is a magical file)

in terminal 2:
bash-4.3# cat /var/run/netns/testnet
bash-4.3#
(shows this is just an empty file)

Revision history for this message
Dave Garrod (dgarrod) wrote :

Thank you all. Now that you have confirmed that this is a true bug/deficiency please could you tell me:

1) If it will be fixed - if so when and in which version.

2) In the meantime is there a workaround I could use? In my specific case I have three SNAP commands in a SNAP. Let's call them A, B and C. It is command A that sets up the network namespace and A will ALWAYS be run before any instances of B or C. Given that is there any magic I can do in B and C to get that network namespace properly mounted in in /run/netns?

Thanks,

Dave

Revision history for this message
Stéphane Graber (stgraber) wrote :

The most likely solution to this would be the introduction of a flag to allow "apps" within the snap snap to share mount namespace. This would be opt-in and would limit what kind of interface you can use from those apps but would solve your problem.

Until then, you could go the lxd way, which is that instead of starting two separate daemons, we're currently starting just one, which executes a wrapper script that runs both of them, in the same mount namespace.

If that's not an option for you, then you'd need to have B and C in your example, run through a small C wrapper that grabs the reference to the mount namespace used by A and setns() into it.

That option would most likely require you to install in devmode as it would need access to other processes and the ability to escape the mount namespace through setns.

Revision history for this message
Dave Garrod (dgarrod) wrote :

Thanks. We can't really do the "lxd" way. The reason being is that "B" and "C" are ephemeral and need to be run multiple times for short periods of time. "A" is the long running "daemon" type process.

I actually would like more than what I'm asking for here. I'd like to know if I can share a "netns" between SNAPs, not just within a SNAP.

In particular I'd like SNAP X to be able to create the namespace and then have a mechanism where SNAP Y can fire off processes in that namespace. In particular SNAP X will be putting particular network devices into a namespace "NNN". I'd then like SNAP YYY to be able to run a process in the context of namespace "NNN" so it can access the network devices in the kernel that are in the "NNN" namespace. Is this going to be possible? This sounds to me something like a "slot/plug" thing. Shouldn't SNAP X be able to declare that namespace "NNN" is a "slot" that is accessible through a "plug" from SNAP Y?

I'm sorry if the question doesn't make sense, we don't have a lot of experience with the whole SNAP model so it is all new to us.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Stephane and I talked about this on IRC and I'll summarize the issue here and some ideas for possible solutions that the snappy team can work through the design on

* snap-confine sets up a private mount namespace for each command invocation (whether daemon or cli)
* the private mount namespace is needed to implement a private /tmp for application isolation instead of setting TMPDIR like we did at first in 15.04 (unfortunately far too many applications don't honor TMPDIR)
* the private mount namespace is needed for the interfaces.SecurityMount backend to function properly. As an example, without a private mount namespace, the content interface would be available to all commands within a snap even if only one command used 'plugs: [ content ]'

Here are a few broadstrokes at options:

1. Do nothing and require developers to do all their namespace setup in one snap command. This option marks this bug as "Won't Fix"

2. Revert the use of private mount namespaces and go back to using TMPDIR and make any interfaces that use the interfaces.SecurityMount backend a 'global interface' where that interface applies to all commands. Unfortunately there is a lot of software that hard codes /tmp which was demonstrated by ROS and the early days of snappy

3. Revert the use of private mount namespaces and implement the preload library for accesses to /tmp and make any interfaces that use the interfaces.SecurityMount backend a 'global interface' where that interface applies to all commands. This is better than '1' and we want the preload library for other reasons, but the 'global interface' requirement is awkward

4. Keep the use of private mount namespaces but make all commands within the same snap share the same mount namespace. This would fix a longstanding bug with /tmp not being shared between commands, but still requires the interfaces.SecurityMount backend be a 'global interface'

5. Retain the current behavior by default but add opt-in yaml for sharing the mount namespace at the global snap level. This is '4' but opt-in.

6. Retain the current behavior by default but add opt-in yaml for sharing the mount namespace at the global snap and app level, possibly with defining arbitrary shared mount namespaces (eg, daemon1 and daemon2 get one mount namespace while daemon3 and command4 get another).

Stephane and I both feel that '1' and '2' leave developer pain points. We feel '6' is overly complex and hard to justify with realistic use cases. '3' has the potential to avoid future pain points due to private mounts at the cost of requiring a preload for every app invocation. '5' allows for the interfaces.SecurityMount backend to operate on the app level but means interfaces like 'content' either sometimes are global (confusing) or cannot be specified with shared mount namespaces (a developer pain point). '4' is simpler to implement/understand in some ways, but the unconditional 'global interface' may not always be desirable.

In our opinion, something along the lines of either 4 or 5 is likely best. Stephane, please correct me if I mispoke.

Revision history for this message
Stéphane Graber (stgraber) wrote :

An extra note, depending on a preload library would also be problematic in that anything which is statically built (hello Golang!) will not be affected by an LD_PRELOAD.

Revision history for this message
John Lenton (chipaca) wrote :

Just FTR I think 4 is what we want; otherwise we'll be doing something awkward to fix the non-shared /tmp issue (and a shared /tmp, which we want, implies the content interface is shared in practice)

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

"a shared /tmp, which we want, implies the content interface is shared in practice" -- yes, and thinking about this overnight I think we could address this in documentation well enough.

FYI, going to talk with Gustavo about this soon for a decision.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Gustavo (snappy architect) and I discussed this at length and decided that the path forward should be option '4'. Ie, unconditionally use a shared mount namespace for all commands within a snap, and document that plugging the content interface will make the content available to all commands within a snap. This will solve this devmode bug, will make use of 'mount' within snaps behave consistently (for interfaces that support using 'mount', eg, lxd and docker) and leaves open the possibility of sharing namespaces between snaps in some manner down the road (as Dave requested).

We briefly discussed implementation ideas to identify potential pitfalls to watch out for (namely, don't accidentally enter another process' mount namespace) and we agreed I would talk to Jamie Bennett (snappy team manager) on getting this work assigned.

Changed in snappy:
status: Confirmed → Triaged
summary: - Cannot share a namespace between apps in a SNAP
+ Cannot share a namespaces created with 'ip netns' between apps in a SNAP
summary: - Cannot share a namespaces created with 'ip netns' between apps in a SNAP
+ Cannot share a namespaces created with 'ip netns' between apps in a
+ devmode SNAP
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Jamie Bennett assigned Zygmunt to the bug (thanks!)

@Zygmunt, once you've decided on a design for the implementation, can you discuss it with Gustavo and the security team? Thanks!

Changed in snappy:
assignee: nobody → Zygmunt Krynicki (zyga)
Revision history for this message
Dave Garrod (dgarrod) wrote : RE: [Bug 1611444] Re: Cannot share a namespace between apps in a SNAP

Sorry for the delay in replying. I was on vacation yesterday.

I absolutely agree with your choice of solution. I think all commands in a SNAP should share an identical environment, especially file directories such as /tmp. I find it strange that you set them up with different environments in the first place i.e. a different private mount namespace for each command within a SNAP.

In regards to interfaces I'm having difficulty finding any information on the "interfaces.SecurityMount" interface. It is not listed when I do a:

snap interfaces

nor can I find it referenced in any documentation. Specifically:

https://developer.ubuntu.com/en/snappy/guides/interfaces/

Also I couldn't find any reference to what you are calling the "Content" interface. That doesn’t seem to be listed on the page referenced above either.

Also please could you explain how in the future I'll be able to create a network namespace in one SNAP and put network interfaces into it (I know how to do this bit, we do it today) and then ACCESS these network interfaces from another SNAP. This will of course involve creating a process inside that second SNAP inside the designated network namespace. This is something we'll absolutely need to to do.

I can't see how this fix will aid us in doing that. Please could you explain.

Thanks,

Dave

________________________________

DISCLAIMER:
This e-mail and any attachments to it may contain confidential and proprietary material and is solely for the use of the intended recipient. Any review, use, disclosure, distribution or copying of this transmittal is prohibited except by or on behalf of the intended recipient. If you have received this transmittal in error, please notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed.

Revision history for this message
David Callé (davidc3) wrote :

@Dave Hi, most of the documentation on snaps is moving to snapcraft.io, the complete interfaces reference can be found here: http://snapcraft.io/docs/reference/interfaces

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

@Dave

"In regards to interfaces I'm having difficulty finding any information on the 'interfaces.SecurityMount' interface."

This is an interface backend and something the interfaces themselves might use. This is an implementation detail of snapd and not something a developer needs to worry about. Though if curious, you can look at the code in snapd.git/interfaces/*.

(skipping the question David answered)

"Also please could you explain how in the future I'll be able to create a network namespace in one SNAP and put network interfaces into it (I know how to do this bit, we do it today) and then ACCESS these network interfaces from another SNAP. This will of course involve creating a process inside that second SNAP inside the designated network namespace. This is something we'll absolutely need to to do."

You are right that using a shared mount namespace per-snap does not directly help with sharing namespaces between snaps. The implementation for fixing this bug should be forward looking for when we want to design interfaces for sharing mount namespaces between snaps. Once we have the implementation for sharing mount namespaces robustly, we should have all the knowledge needed to design and implement support for sharing mount namespaces between snaps.

Revision history for this message
Dave Garrod (dgarrod) wrote : RE: [Bug 1611444] Re: Cannot share a namespaces created with 'ip netns' between apps in a devmode SNAP

Thank you for the information.

Re:

> You are right that using a shared mount namespace per-snap does not directly help with sharing namespaces between snaps. The implementation
> for fixing this bug should be forward looking for when we want to design interfaces for sharing mount namespaces between snaps. Once we have
> the implementation for sharing mount namespaces robustly, we should have all the knowledge needed to design and implement support for sharing
> mount namespaces between snaps.

I think I'm getting a bit confused between when you talk about the "shared mount namespace" and the specific issue regarding "a network (NETNS) namespace"
that is the subject of this bug report.

My understanding that all you are doing here for now is sharing the "private mount" namespace between commands inside a SNAP. As a side effect of this the "directory" (/run/netns/NAME) that the network namespace gets mounted in ends up being shared between all commands in the SNAP and this the "ip netns exec NAME COMMAND" will now work in the context of all commands in a SNAP.

Since I'm sure you won't be sharing the whole "private mount" namespace between SNAPs I don't see how this is going to help me access network devices created in a network namespace from one SNAP by another SNAP.

As far as I can see the current interfaces:

:network
:network-bind
:network-control
:network-manager
:network-observe

Are what are used to enable a SNAP to access network devices in the default namespace. I guess what I am looking for is a similar mechanism that does the name thing but instead proves accesses to interfaces in a named network namespace. In addition I guess the second SNAP needs to be able to launch )or internally fork a process in the designated network namespace. But to do this from what you have described so far will involve this SNAP somehow having to gain access to the /run/netns/NAME special file that has been mounted in the private namespace of the SNAP that created that network namespace. How is this going to work. Just as importantly to me I'd like to know is there a release planned where I will be able to do this.

Thanks again,

Dave

PS Let me know if you'd prefer I break the INTER SNAP NETNS discussion off into another bug or even the listserv. I'm still at the stage where I'm busily reading and trying to understand the whole SNAP infrastructure and some of the philosophy behind it hence some of my more general inquiries.

________________________________

DISCLAIMER:
This e-mail and any attachments to it may contain confidential and proprietary material and is solely for the use of the intended recipient. Any review, use, disclosure, distribution or copying of this transmittal is prohibited except by or on behalf of the intended recipient. If you have received this transmittal in error, please notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed.

Revision history for this message
Mark Shuttleworth (sabdfl) wrote : Re: [Bug 1611444] Re: Cannot share a namespace between apps in a SNAP

On 19/08/16 11:13, Dave Garrod wrote:
> I absolutely agree with your choice of solution. I think all commands in
> a SNAP should share an identical environment, especially file
> directories such as /tmp. I find it strange that you set them up with
> different environments in the first place i.e. a different private mount
> namespace for each command within a SNAP.

I think the rationale was to do with cases where a particular command
needs access to special content provided by a different snap, which is
sensitive somehow.

So, you have a snap which has SecretFiles and then you have other snaps
with commands that need access to SecretFiles.... but not EVERY command
needs that access. Using different mount namespaces was a convenient way
to ensure that only specific commands get such access. But I think that
using a common mount namespace is much cleaner, and other mechanisms
like apparmor or selinux can provide access control to such content
per-command.

I think consolidating to a common mount namespace is correct, but the
backstory is interesting because it illustrates the way the snap folks
are aiming to set the highest possible bar for security in snaps, which
is great.

Mark

Revision history for this message
Mark Shuttleworth (sabdfl) wrote : Re: [Bug 1611444] Re: Cannot share a namespaces created with 'ip netns' between apps in a devmode SNAP

On 19/08/16 12:40, Dave Garrod wrote:
> My understanding that all you are doing here for now is sharing the
> "private mount" namespace between commands inside a SNAP. As a side
> effect of this the "directory" (/run/netns/NAME) that the network
> namespace gets mounted in ends up being shared between all commands in
> the SNAP and this the "ip netns exec NAME COMMAND" will now work in the
> context of all commands in a SNAP.
>
> Since I'm sure you won't be sharing the whole "private mount" namespace
> between SNAPs I don't see how this is going to help me access network
> devices created in a network namespace from one SNAP by another SNAP.
>
> As far as I can see the current interfaces:
>
> :network
> :network-bind
> :network-control
> :network-manager
> :network-observe
>
> Are what are used to enable a SNAP to access network devices in the
> default namespace. I guess what I am looking for is a similar mechanism
> that does the name thing but instead proves accesses to interfaces in a
> named network namespace. In addition I guess the second SNAP needs to be
> able to launch )or internally fork a process in the designated network
> namespace. But to do this from what you have described so far will
> involve this SNAP somehow having to gain access to the /run/netns/NAME
> special file that has been mounted in the private namespace of the SNAP
> that created that network namespace. How is this going to work. Just as
> importantly to me I'd like to know is there a release planned where I
> will be able to do this.

Yes... where at a tricky intersection of mount namespaces and network
namespaces. Clearly what you want is useful, clearly the fact that the
filesystem is (magically) involved is awkward :)

I think there is value in having a slot interfaces that allows one snap
with the slot to manufacture network namespaces that other snaps with
the matching plug can consume. Exactly how we make that happen is
interesting.

As for timelines, I can say your particular project is important to us
so we'll keep focused on it till we get you a way forward :)

Mark

Zygmunt Krynicki (zyga)
Changed in snap-confine:
milestone: none → 1.0.41
assignee: nobody → Zygmunt Krynicki (zyga)
importance: Undecided → Critical
Revision history for this message
Zygmunt Krynicki (zyga) wrote :

I'm catching up with all the discussion with the goal to come up with a plan and implement support for this feature.

Changed in snap-confine:
status: New → Fix Committed
status: Fix Committed → In Progress
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

@Zygmunt, note that this bug is only about sharing the mount namespace between apps within a snap.

@Dave, I will be looking at options for enabling the use of the 'ip netns exec <namespace> ...' command as a separate issue. To make it clear what is happening-- going into a network namespace with setns() is not blocked by the mount namespace. What is happening is 'ip netns exec' is meant to be used with applications that are network-namespace-unaware and it creates a mount namespace on the fly and bind mounts various files in /etc so the application 'just works' unmodifed. 'man ip-netns' has more information.

Revision history for this message
Dave Garrod (dgarrod) wrote :
Download full text (3.5 KiB)

Re:

> @Dave, I will be looking at options for enabling the use of the 'ip netns exec <namespace> ...' command as a separate issue. To make it clear what is happening-- going into a
> network namespace with setns() is not blocked by the mount namespace. What is happening is 'ip netns exec' is meant to be used with applications that are
> network-namespace-unaware and it creates a mount namespace on the fly and bind mounts various files in /etc so the application 'just works' unmodifed.
> 'man ip-netns' has more information.

Inter SNAP it is not the use of the "ip netns exec <namespace> ..." command that I'm most interested in. What I need to be able to do is either:

a) Fire up the SNAP already running in a network namespace that was defined by another SNAP
or
b) From within the SNAP either form off a process in the specified network namespace or move the process to the specified network namespace.

As you point out once the process is running in the context of the specified namespace it will be able to access the kernel "network devices" defined in that namespace and of course not have access to the kernel "network devices" that are still in the default namespace.

Dave

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Jamie Strandboge
Sent: Monday, August 22, 2016 10:16 AM
To: David Garrod
Subject: [Bug 1611444] Re: Cannot share a namespaces created with 'ip netns' between apps in a devmode SNAP

@Zygmunt, note that this bug is only about sharing the mount namespace between apps within a snap.

@Dave, I will be looking at options for enabling the use of the 'ip netns exec <namespace> ...' command as a separate issue. To make it clear what is happening-- going into a network namespace with setns() is not blocked by the mount namespace. What is happening is 'ip netns exec'
is meant to be used with applications that are network-namespace-unaware and it creates a mount namespace on the fly and bind mounts various files in /etc so the application 'just works' unmodifed. 'man ip-netns'
has more information.

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1611444

Title:
  Cannot share a namespaces created with 'ip netns' between apps in a
  devmode SNAP

Status in Snappy Launcher:
  In Progress
Status in Snappy:
  Triaged

Bug description:
  Please see:

  https://<email address hidden>/msg00542.html

  for additional details. It was requested that I move that discussion
  to this bug report.

  In summary it appears that multions "apps" in a SNAP cannot share the
  same NETNS namespace. If one app create a namespace the other apps in
  SNAP cannot use it. They get assorted errors like:

  RTNETLINK answers: Invalid argument

  Please see the details in the mail archive posting.

To manage notifications about this bug go to:
https://bugs.launchpad.net/snap-confine/+bug/1611444/+subscriptions

________________________________

DISCLAIMER:
This e-mail and any attachments to it may contain confidential and proprietary material and is solely f...

Read more...

Revision history for this message
Zygmunt Krynicki (zyga) wrote :
Revision history for this message
Zygmunt Krynicki (zyga) wrote :

This is now implemented but there's a corresponding apparmor bug that may cause this to fail.

Changed in snap-confine:
status: In Progress → Fix Committed
Revision history for this message
Zygmunt Krynicki (zyga) wrote :

The workaround that makes the apparmor bug affecting this go away is in the network-control interface so snaps using that should work just fine

Changed in snappy:
status: Triaged → Fix Committed
status: Fix Committed → Invalid
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

To be clear, this is the apparmor bug: https://launchpad.net/bugs/1624497 and this is the workaround for devmode snaps:

1. be sure to specify 'plugs: [network-control]' in snapcraft.yaml
2. Install in devmode like normal. Eg: sudo snap install --devmode /path/to/your/snap
3. Connect the network-control interface. Eg: sudo snap connect SNAP_NAME:network-control ubuntu-core:network-control

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Also, now that this is implemented, we can start working on a network-namespace interface: https://bugs.launchpad.net/snappy/+bug/1624675

tags: added: snapd-interface
Zygmunt Krynicki (zyga)
Changed in snap-confine:
status: Fix Committed → Fix Released
Zygmunt Krynicki (zyga)
description: updated
Changed in snap-confine (Ubuntu):
status: New → Fix Released
Changed in snap-confine (Ubuntu Xenial):
status: New → In Progress
Revision history for this message
Leo Arias (elopio) wrote :

I ran the snap-confine test in an up-to-date xenial classic kvm, after enabling proposed and upgrading to snap-confine to 0.43.

I got no errors, looks good.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

@Dave, just to give you a quick update since quite a lot has happened in this area but that may not be apparent if you are tracking 16.04.

 * snap-confine 1.0.44 was updated such that every command with a snap shares the same mount space. That fixes this bug so that different commands in a devmode snap can use 'ip netns'
 * snapd 2.17 introduced the concept of 'bidirectional mounts' which allows a snap to propagate changes to a mount point to the global namespace or other snaps, which laid the groundwork for your use cases as mentioned in comment #29
 * I've built upon this and started working on bug #1624675 for allowing different snaps to create/delete/manage network namespaces with 'ip netns' (as well as use 'ip netns exec' and also setns() to enter existing network namespaces). I'm hoping we can get this in snapd 2.20.

I suggest you also subscribe to bug #1624675.

If you are tracking 16.04, snapd 2.17.1 and snap-confine 1.0.44 are in xenial-proposed and awaiting confirmation to be released to xenial-updates. This bug will be fixed with those updates.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

FYI, bug #1624675 is now fixed in master via the 'network-control' interface and will be available in snapd/snap-confine 2.20. During PR review, Gustavo and I agreed that this didn't need to be broken out into a separate interface after all.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Marking fix released since 2.20 is available in 16.04 now.

Changed in snap-confine (Ubuntu Xenial):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.