dhclient fails for virtual interfaces (IP aliases)

Commenting on some of the comments in Bug #123773.

In Bug #123773
Jamie Strandboge wrote:
> ...though ifup (and ifdown) will give that error, the interface
> comes up fine, and is pingable

In this case the interface doesn't come up.

In comment
https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/123773/comments/11
behemot wrote:
> ...that removing package wireless-tools...helps.

Removing, moving, or disabling the execute bits on /etc/network/if-pre-up.d/wireless-tools seems to have no impact on this problem.

In comment
https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/123773/comments/12
Alessandro Lo Forte wrote:
> I have located the problem in the file /etc/if-pre-up.d/wireless-tools...
> In fact the error code is generated when /sbin/ifconfig is invoked to
> bring up an alias (like eth0:0) interface.

This suggests both problems likely have a common underlying cause. My research shows that /sbin/ifconfig, being invoked by dhclient, is also the culprit here. (See comments below.)

This may also mean that this bug should be reclassified as being against the net-tools package instead of dhcp3.

In comment
https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/123773/comments/22
Juuso Tähkäpää wrote:
> sudo ifconfig eth0:1 1.2.3.4
> dhcpcd -I someid eth0:1
> ...but still ifconfig shows no eth0:1.
> Manually assigned aliases (ifconfig eth0:2 $ip) do show up
> in ifconfig listings just as they should.

Although I don't have dhcpcd installed, I've observed similar results with dhclient, where it fails to configure the interface, but invoking ifconfig directly with a static address works.

This gives some hope for a possible workaround hack. I'm thinking in /etc/network/interfaces one could invoke a script using the 'up' argument for the main (real) interface, which then would invoke dhclient in some fashion or maybe using a simpler DHCP client like udhcpc that returns an IP address to a shell variable, and then invokes ifconfig, passing the address. It would have limitations, like requiring perpetual leases (I think, as there would be no process left running to handle an expired lease).

In comment
https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/123773/comments/17
gumptravels wrote:
> ...makes me think that you may not be able use dhcp on virtual interfaces
> since the mac address would be the same as all the other virtual interfaces
> on that same parent, and the same as the parent itself.

While a MAC address is often used as the default way to identify a DHCP client, the DHCP client can optionally supply other identifying information. For example, this theoretically correct (untested, due to this bug) stanza from /etc/network/interfaces specifies that when the DHCP client requests an address for the virtual interface that it identifies itself with the alternate host name, "indianpoint."

auto eth0:1
iface eth0:1 inet dhcp
  hostname indianpoint

Being able to use DHCP on virtual interfaces is highly desirable in order to keep maintenance to a minimum, especially if you are using an integrated DHCP/DNS service, like Dnsmasq. In that case the act of requesting an address with the alternate host name will automatically ...

Forum threads relating to this issue:
http://ubuntuforums.org/showthread.php?t=918015
http://ubuntuforums.org/showthread.php?t=515162

Attached is an archive containing the output of:

  # strace -ff -F -o /tmp/dhclient.strace dhclient eth0:0
  There is already a pid file /var/run/dhclient.pid with pid 134519072
  Internet Systems Consortium DHCP Client V3.0.6
  Copyright 2004-2007 Internet Systems Consortium.
  All rights reserved.
  For info, please visit http://www.isc.org/sw/dhcp/

  SIOCSIFADDR: Permission denied
  SIOCSIFFLAGS: Permission denied
  SIOCSIFFLAGS: Permission denied
  Bind socket to interface: No such device

Some interesting bits include:

From dhclient.strace.9271-avahi-autoipd:

execve("/usr/sbin/avahi-autoipd", ["/usr/sbin/avahi-autoipd", "-k", "eth0:0"], [/* 7 vars */]) = 0
[...]
open("/var/run//avahi-autoipd.eth0:0.pid", O_RDWR) = -1 ENOENT (No such file or directory)
write(2, "Failed to kill daemon: No such f"..., 48) = 48
write(2, "\n", 1) = 1
exit_group(1) = ?

That avahi-autoipd is trying to access /var/run/avahi-autoipd.eth0:0.pid may be by design, or may be a bug. I don't know enough about avahi to know whether it normally has a separate daemon running for each interface, and if so, whether that should apply to virtual interfaces.

I saw a similar error in some of the messages output by dhclient, where it was trying to create a PID file for itself that incorporated the virtual device name. Again, should it be running a separate instance of the daemon for each virtual interface?

And from dhclient.strace.9272-ifconfig:

execve("/sbin/ifconfig", ["ifconfig", "eth0:0", "inet", "0", "up"], [/* 7 vars */]) = 0
[...]
socket(PF_FILE, SOCK_DGRAM, 0) = 3
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
access("/proc/net/if_inet6", R_OK) = 0
socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 6
[...]
ioctl(4, SIOCSIFADDR, 0xbfdbeb00) = -1 EACCES (Permission denied)
dup(2) = 7
fcntl64(7, F_GETFL) = 0x2 (flags O_RDWR)
fstat64(7, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 10), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f0b000
_llseek(7, 0, 0xbfdbe498, SEEK_CUR) = -1 ESPIPE (Illegal seek)
write(7, "SIOCSIFADDR: Permission denied\n", 31) = 31
close(7) = 0
munmap(0xb7f0b000, 4096) = 0
ioctl(4, SIOCGIFFLAGS, {ifr_name="eth0:0", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(4, SIOCSIFFLAGS, 0xbfdbe9e8) = -1 EACCES (Permission denied)
dup(2) = 7
fcntl64(7, F_GETFL) = 0x2 (flags O_RDWR)
fstat64(7, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 10), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f0b000
_llseek(7, 0, 0xbfdbe438, SEEK_CUR) = -1 ESPIPE (Illegal seek)
write(7, "SIOCSIFFLAGS: Permission denied\n", 32) = 32
close(7) = 0
munmap(0xb7f0b000, 4096) = 0
ioctl(4, SIOCGIFFLAGS, {ifr_name="eth0:0", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(4, SIOCSIFFLAGS, 0xbfdbe9e8) = -1 EACCES (Permission denied)
dup(2) = 7
fcntl64(7, F_GET...

SIOCSIFFLAGS: Cannot assign requested address
SIOCSIFFLAGS: Cannot assign requested address

These two errors are because dhclient-script does not support alias interfaces. If you don't use this script, those errors go away. However it's likely dhclient doesn't support alias interfaces at all, a different dhcp client may the solution.

# dhclient eth0:0 -sf /bin/true
Internet Systems Consortium DHCP Client V3.1.1
Copyright 2004-2008 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Bind socket to interface: No such device

I believe this is a missing feature of ISC's dhclient. I can't find any mention in the source of supporting virtual/alias interfaces outside of the alias declaration in dhclient.conf(5) that allows you to specify a static address to be applied to an alias interface in addition to the address acquired by DHCP.

I did test and confirm that dhcpcd=1:3.2.3-1.1 does work on alias interfaces on intrepid.

Bryan McLellan wrote:
> These two errors are because dhclient-script does not support alias interfaces.

That may be partly right. I see /sbin/dhclient-script contains the comment:

# The alias handling in here probably still sucks. -mdz

and there's clearly evidence that it is attempting to support virtual interfaces:

        if [ -n "$alias_ip_address" ]; then

But real problem seems to be that the script contains lines like:

        ifconfig $interface inet 0 up

and the current version of ifconfig spits out those errors when a virtual interface is set to a zero IP address. The evidence suggests that this worked at one time. In fact, multiple DHCP packages seem to expect this operation to work. I recently tried udhcpc and got familiar results:

# udhcpc --hostname=indianpoint --interface=eth0:0 --pidfile=/var/run/udhcpc.eth0:0.pid
udhcpc (v0.9.9-pre) started
SIOCSIFFLAGS: Cannot assign requested address
Sending discover...
Sending select for 192.168.0.235...
Lease of 192.168.0.235 obtained, lease time -1
Resetting default routes
adding dns 192.168.0.35

In this case the error is triggered when udhcpc tries to "deconfigure" the interface before it configures it, but it happily ignores the error.

I'm wondering if one workaround might be to modify the scripts to do:
    ifconfig $interface down
if $interface is virtual. (It seems dhclient-script already contains code to detect virtual interfaces.) I presume there is a reason why "deconfiguring" an interface is normally performed instead of downing it. (I'm guessing it might make a difference if you were running other networking protocols in addition to IP.) The distinction may be irrelevant for a virtual interface, or just for the vast majority of use cases.

Running:
ip addr del $IP dev eth0:0

seems to work too, but has the effect of bringing down the virtual interface and you need extra code to obtain the current IP address (passing in a zero does nothing).

Before a permanent fix can be determined, it needs to be understood whether this represents a bug in ifconfig, or an intentional change in its behavior. I'm thinking of opening up a ticket against ifconfig that this bug can be dependent on.

> If you don't use this script, those errors go away.

But is that a practical option? Doesn't the script provide a pile of needed glue?

Patching the script seems more viable.

> I can't find any mention in the source of supporting virtual/alias interfaces...

The code in dhclient-script is enough to convince me that it was the intention to support virtual interfaces. I'm pretty sure I've read reports that they used to work prior to Ubuntu 8.04.

> I did test and confirm that dhcpcd=1:3.2.3-1.1 does work on alias interfaces on intrepid.

Is it a drop-in replacement that works with the rest of the existing infrastructure in Ubuntu, like /etc/network/interfaces?

My next step was going to be using the 'up' argument in an interface stanza for a physical interface to call udhcpc to configure a virtual interface. It looks like this will work, but using something that works with the normal documented way of declaring virtual interfaces in /etc/network/interfaces is preferable.

On Thu, Apr 2, 2009 at 11:20 PM, Tom Metro <email address hidden> wrote:
> That may be partly right. I see /sbin/dhclient-script contains the
> comment:
>
> # The alias handling in here probably still sucks. -mdz
>
> and there's clearly evidence that it is attempting to support virtual
> interfaces:
>
> if [ -n "$alias_ip_address" ]; then

dhclient-script(8) says "If an IP alias has been declared in
dhclient.conf, its address will be passed in $alias_ip_address, and
that ip alias should be deleted from the interface, along with any
routes to it." Most likely that comment and surely that code is to
support the inclusion of an alias definition in your dhclient.conf(5),
as opposed to running dhclient on an alias interface.

>> If you don't use this script, those errors go away.
>
> But is that a practical option? Doesn't the script provide a pile of
> needed glue?
>
> Patching the script seems more viable.

The script does provide a lot of glue, but working around it as a
debugging technique isolates what errors are coming from the script
and what errors are coming from the program itself. I don't believe
patching the script will do you any good until the 'Bind socket to
interface: No such device' error from dhclient is resolved, because it
appears to be unable to use the interface to send the DHCP request.

>> I did test and confirm that dhcpcd=1:3.2.3-1.1 does work on alias interfaces on intrepid.
>
> Is it a drop-in replacement that works with the rest of the existing
> infrastructure in Ubuntu, like /etc/network/interfaces?

Removing dhcp3-client (which removed ubuntu-minimal) and installing
dhcpcd worked just fine with /etc/network/interfaces on a test vm for
me about five minutes ago, and thus appears like a clean enough work
around.

Bryan McLellan wrote:
> Most likely that comment and surely that code is to
> support the inclusion of an alias definition in your dhclient.conf(5),
> as opposed to running dhclient on an alias interface.

Ah, right. You're probably right.

The dhclient.conf(5) man page also mentions pseudo interfaces, which
sounds a lot like virtual interfaces:

   Under some circumstances it can be useful to declare a
   pseudo-interface and have the DHCP client acquire a configuration for
   that interface. ... A pseudo-interface is just another state machine
   running on the interface named real-name, with its own lease and its
   own state.

It also notes:

   You must also provide a separate client script for the
   pseudo-interface to do what you want with the IP address.

If by that they mean your script will be ran in place of
/sbin/dhclient-script, then that would be some more evidence supporting
your point that /sbin/dhclient-script doesn't support virtual
interfaces, as is.

>> Patching the script seems more viable.
>
> The script does provide a lot of glue, but working around it as a
> debugging technique isolates what errors are coming from the script
> and what errors are coming from the program itself.

I guess I was already pretty convinced that the source of the errors was
ifconfig, given that the error can be reproduced external to dhclient
and it's scripts by invoking ifconfig directly using the same arguments.
The next questions is: is ifconfig broken, or is the manner in which
dhclient is calling it broken?

> I don't believe patching the script will do you any good until the
> 'Bind socket to interface: No such device' error from dhclient is
> resolved...

My assumption was that this was simply a side effect of the
"deconfigure" step failing, which results in the virtual interface never
being created, and thus dhclient has no device to work with.

This can be resolved by either 1. patching the code so that it
instantiates the virtual interface in a deconfigured state, or 2.
special casing virtual interfaces so that the physical interface gets
used for the DHCP handshake, while the specified virtual interface is
used for the resulting call to ifconfig. (Apparently the latter happens
with udhcpc.)

> Removing dhcp3-client (which removed ubuntu-minimal)...

Removing ubuntu-minimal sounds like something to be avoided, but there
was mention of people doing that (as a side effect of removing the
wireless package) in the bug this one was split off from, and it didn't
sound like it caused any problems. I'll have to take a look at
ubuntu-minimal to see what it is.

> ...and installing dhcpcd worked just fine with
> /etc/network/interfaces ... and thus appears like a clean enough work
> around.

Excellent. Thanks for the suggested workaround. Care to post the
/etc/network/interfaces syntax you used for the virtual interface? (In
my research I ran across about 3 or 4 variations.)

On Fri, Apr 3, 2009 at 1:20 PM, Tom Metro <email address hidden> wrote:
> Removing ubuntu-minimal sounds like something to be avoided, but there
> was mention of people doing that (as a side effect of removing the
> wireless package) in the bug this one was split off from, and it didn't
> sound like it caused any problems. I'll have to take a look at
> ubuntu-minimal to see what it is.

It's a meta package, like installing 'ubuntu-desktop' will install all
of the packages you need for a standard Ubuntu desktop, -minimal will
give you a basic system. Uninstalling it would mean that in the future
if another package was added to to the meta package, you wouldn't get
that additional package on upgrade. You can run 'dpkg-query -f
'${Depends}\n' -W ubuntu-minimal' to see the packages that it
installs. You could also remove the dhcp3-client package with
--force-depends, which just runs the risk that it gets reinstalled
later during an upgrade. There's probably a complex pinning solution
too, but I can't fathom it.

> Excellent. Thanks for the suggested workaround. Care to post the
> /etc/network/interfaces syntax you used for the virtual interface? (In
> my research I ran across about 3 or 4 variations.)

auto eth0
iface eth0 inet dhcp

auto eth0:0
iface eth0:0 inet dhcp

I've opened Bug #356800, "ifconfig fails to initiaize a virtual interface with a zero IP address."

This bug should be made dependent on it. (I didn't see a way to do that.)

Bryan McLellan wrote:
> Tom Metro wrote:
>> I'll have to take a look at ubuntu-minimal to see what it is.
>
> It's a meta package...

I figured that from the name, but I wondered what packages it pulled in,
and what the consequences were for removing it were.

> Uninstalling it would mean that in the future if another package was
> added to to the meta package, you wouldn't get that additional
> package on upgrade.

Which sounds fairly low risk, but something one would ideally want to avoid.

> You could also remove the dhcp3-client package with
> --force-depends, which just runs the risk that it gets reinstalled
> later during an upgrade. There's probably a complex pinning solution
> too, but I can't fathom it.

I couldn't figure out a way around it. Any combination I came up with
(like putting dhcp3-client on hold, then removing it), just left the
package system in a broken state.

More importantly, on Hardy there are additional dependencies on
dhcp3-client. dhcdbd depends on it, which provides the DBus interface to
dhcp3-client that's used by network manager, which in turn Network
Manager depends on, which in turn ubuntu-desktop depends on.

The resolutions offered up by aptitude (supposedly the best tool for
resolving dependency issues) all involved uninstalling more packages
than I cared to.

To work around this I ended up creating a dummy package using the equivs
package. That lets me replace the real dhcp3-client with a dummy one,
which uninstalled the dhcp3-client binaries and scripts.

All this makes me wonder if ubuntu-minimal ought to be dependent on the
generic dhcp-client (which dhcp3-client, dhcpcd, and udhcpc all
provide), rather than a specific package. (Of course on Hardy that won't
fix the dhcdbd dependency. I'm guessing that means dhcp3 is probably the
only client Network Manager knows how to control.)

I'm also wondering if uninstalling dhcp3-client was even necessary. I
noticed when I installed udhcpc that it didn't cause any conflicts, even
though it ought to have conflicted with another package providing
dhcp-client. Either udhcpc isn't correctly tagged as providing
dhcp-client, or the two can peacefully coexist.

Looks like dhcpcd has dhcp-client listed as a conflict, while udhcpc
doesn't. Given that they all are driven by the sane interfaces config
file, I'd expect there to be conflicts. I'd like to understand better
how that works. Is each overwriting the other's ifupdown scripts? Is
there a common script with a bunch of -x tests to see which binaries are
present?

> I did test and confirm that dhcpcd=1:3.2.3-1.1 does work on alias
> interfaces on intrepid.

Unfortunately this doesn't work on Hardy with dhcpcd=1:3.0.17-2:

# init.d/networking restart
                                       * Reconfiguring network interfaces...
Error, eth0: dhcpcd not running
dhcpcd.sh: interface eth0 has been configured with new IP=192.168.0.203
  * Stopping NTP server ntpd
    ...done.
  * Starting NTP server ntpd
    ...done.
Error, eth0:1: ioctl SIOCSIFFLAGS: Cannot assign requested address
Failed to bring up eth0:1.
[...]
    ...done.

Looks like the usual failure, though in a bit different format than what
ifconfi...

Here's the summary how how to work around this bug:

First, if you're using Dnsmasq on the server-side you might run into Bug #327703. The Dnsmasq author has a patched version available.

1. On Hardy, download and install the attached dhcp3-client dummy package:
 % sudo dpkg -i dhcp3-client-dummy_3.0.6.dfsg-1ubuntu9_all.deb
and then put it on hold:
 % sudo aptitude hold dhcp3-client
On Intrepid, remove the dhcp3-client package and permit ubuntu-minimal to also get removed (or build your own dummy package using the equivs package).

2. Install udhcpc:
 % sudo aptitude install udhcpc

3. Add your virtual interface to /etc/network/interfaces, while also adding unique hostname and client arguments for both. The client identifiers can match the corresponding host names or be any other reasonable string, as long as they are not used by any other DHCP clients on your network.

auto eth0
iface eth0 inet dhcp
  hostname primary-host
  client primary-client-id

auto eth0:1
iface eth0:1 inet dhcp
  hostname virtual-host
  client virtual-client-id

4. Restart the network subsystem:
 % sudo /etc/init.d/networking restart

Confirming this bug is still present on Ubuntu 10.10 desktop and 10.04 server.

and on Ubuntu 11.04 desktop. It does not look secondary to me that default dhcp client can't work on virtual interfaces. What about you, ubuntu's guys?

Can confirm that it is still in Debian stable as of 2016. Cannot test Ubuntu now. Installing udhcpc and uninstalling isc-dhcp-client does the trick. No need to put any package on hold. At least, when you are still using /etc/network/interfaces. I have not tested network-manager or anything else. If you only have one DHCP address (or even multiple? I cannot test) giving hostname and client is not necessary in /etc/network/interfaces.