DKMS brcompat module circular dependency causes broken module

Bug #1125611 reported by Sebastian Marsching
44
This bug affects 7 people
Affects Status Importance Assigned to Milestone
openvswitch (Ubuntu)
Fix Released
Critical
Unassigned
Precise
Fix Released
Critical
James Page
Quantal
Fix Released
Critical
James Page

Bug Description

[Impact]
brcompat module is broken

[Test Case]
sudo apt-get install openvswitch-datapath-dkms
sudo apt-get install openvswitch-brcompat
edit /etc/default/openvswitch-switch and enable BRCOMPAT=yes
sudo service openvswitch-switch restart

brcompat daemon will fail to start:
ovs-brcompatd: could not open brcompat socket. Check "brcompat" kernel module.

/var/log/syslog:
brcompat: module license 'unspecified' taints kernel.

Note that the 'force-reload-kmod' may be required after upgrading to force the DKMS modules to be reloaded:

sudo service openvswitch-switch force-reload-kmod

[Regression Potential]
Minimal; the additional cherry picked patch renames the brcompat.c file brcompat_main.c to avoid a circular dependency during DKMS build.

[Original bug report]
On Ubuntu 12.04.2 LTS, after an automatic upgrade from openvswitch 1.4.0-1ubuntu1.3 to openvswith 1.4.0-1ubuntu1.4 the brcompat support does not work any longer.

The brcompat kernel module can be build, installed, and loaded, but openvswitch-brcompatd will not start with the error message: "ovs-brcompatd: could not open brcompat socket. Check "brcompat" kernel module."

After a downgrade back to version 1.4.0-1ubuntu1.3 brcompat works again. I think this is essentially the same problem as described here: https://answers.launchpad.net/ubuntu/+source/openvswitch/+question/221334

It might be unrelated, but after the upgrade "lsmod" does not list the "openvswitch" module as a dependency of the "brcompat" module. After downgrading, the dependency is shown again.

Revision history for this message
Thiago Martins (martinx) wrote :

Who is responsible for this? Some compiler bot isn't doing its job? lol

The broken Open vSwitch (1.4.0-1ubuntu1.4) `brcompat' package is now available at proposed-updates repository!!

How can a broken package like this pass through SRU (https://wiki.ubuntu.com/StableReleaseUpdates)?

This was supposed to be a LTS version... Right? =P

The module brcompat doesn't work after the latest upgrade.

Versions:

Linux 3.2.0-38-virtual
Open vSwitch 1.4.0-1ubuntu1.4

Note:

brcompat of Open vSwitch 1.4.0-1ubuntu1.3 is fine.

Linux Bridge module working as a fallback. Open vSwitch 1.4.0-1ubuntu1.4 entirely disabled...

You'll need to disable the `-proposed' and `-updates' APT repositories, purge and reinstall openvswitch-* 1.4.0-1ubuntu1.3 packages...

Best,
Thiago

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvswitch (Ubuntu):
status: New → Confirmed
Revision history for this message
Peng Yong (ppyy) wrote :

confirmed! my openstack zone are broken.

how can a so serious bug got into LTS version?

Revision history for this message
Thiago Martins (martinx) wrote :

Same here... My Openstack with nova-network and Open vSwitch got down until I downgrade the package back to `1.4.0-1ubuntu1.3'.

LTS crash... Something is wrong not only with this openvswitch Ubuntu package but, all future LTS versions must be *-RELOADED... =P

Cheers!
Thiago

Revision history for this message
Peng Yong (ppyy) wrote :

the bug was in 2 month ago, and backport to LTS now:

https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1088160

how is QA team works in Ubuntu LTS?

Revision history for this message
Thiago Martins (martinx) wrote :

Apparently, there is no QA Team.

Both -proposed and -updates repository are disabled for now.

100 * (facepalm)

Revision history for this message
Sebastian Marsching (sebastian-marsching) wrote :

I do not think that it is the same bug as originally described in bug #1088160. This bug was an issue with the build system. However, it might be the same problem that that one comment (https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1088160/comments/2) to that bug described.

James Page (james-page)
Changed in openvswitch (Ubuntu):
importance: Undecided → Critical
Revision history for this message
James Page (james-page) wrote :

Problem identified; circular dependency due to module renaming was causing the brcompat object not to be included in the module (hence the message about tainting the kernel).

Changed in openvswitch (Ubuntu Precise):
status: New → Confirmed
Changed in openvswitch (Ubuntu Quantal):
status: New → Confirmed
Changed in openvswitch (Ubuntu):
status: Confirmed → Fix Released
Changed in openvswitch (Ubuntu Precise):
importance: Undecided → Critical
Changed in openvswitch (Ubuntu Quantal):
importance: Undecided → Critical
Changed in openvswitch (Ubuntu Precise):
assignee: nobody → James Page (james-page)
Changed in openvswitch (Ubuntu Quantal):
assignee: nobody → James Page (james-page)
Revision history for this message
James Page (james-page) wrote :

This commit was missing from the backport of fixes to support 3.5 kernel:

http://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=commit;h=bf8ba92e4875d0ca4c26ddd2a4c46fd26f8e401b

summary: - Upgrade to 1.4.0-1ubuntu1.4 breaks brcompat support
+ DKMS brcompat module circular dependency causes broken module
Changed in openvswitch (Ubuntu Precise):
status: Confirmed → In Progress
Changed in openvswitch (Ubuntu Quantal):
status: Confirmed → In Progress
Revision history for this message
James Page (james-page) wrote :

Fix uploaded to precise-proposed for SRU team review.

I've also uploaded the same fix to https://launchpad.net/~james-page/+archive/openvswitch

description: updated
James Page (james-page)
description: updated
description: updated
Revision history for this message
James Page (james-page) wrote :

Fix also uploaded to quantal-proposed for SRU team review.

Revision history for this message
Sebastian Marsching (sebastian-marsching) wrote :

I can confirm the the new version (from the PPA) fixes the problem for me.

Thank you very much for the quick solution!

Revision history for this message
Peng Yong (ppyy) wrote :
Download full text (4.1 KiB)

error in openstack:

# lsmod|grep brcomp
brcompat 13512 0
openvswitch 84038 8 brcompat

# service openvswitch-switch status
ovsdb-server is running with pid 2186
ovs-vswitchd is running with pid 2228
ovs-brcompatd is running with pid 3007

2013-02-17 10:26:14 DEBUG nova.utils [req-1591a4d4-9779-452f-bd87-2801f25ae592 None None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addif qbreb2557ea-3b qvbeb2557ea-3b execute /usr/lib/python2.7/dist-packages/nova/utils.py:183
2013-02-17 10:26:14 DEBUG nova.utils [req-1591a4d4-9779-452f-bd87-2801f25ae592 None None] Result was 1 execute /usr/lib/python2.7/dist-packages/nova/utils.py:199
2013-02-17 10:26:14 2046 CRITICAL nova [-] Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addif qbreb2557ea-3b qvbeb2557ea-3b
Exit code: 1
Stdout: ''
Stderr: "can't add qvbeb2557ea-3b to bridge qbreb2557ea-3b: Invalid argument\n"
2013-02-17 10:26:14 2046 TRACE nova Traceback (most recent call last):
2013-02-17 10:26:14 2046 TRACE nova File "/usr/bin/nova-compute", line 48, in <module>
2013-02-17 10:26:14 2046 TRACE nova service.wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 659, in wait
2013-02-17 10:26:14 2046 TRACE nova _launcher.wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 192, in wait
2013-02-17 10:26:14 2046 TRACE nova super(ServiceLauncher, self).wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in wait
2013-02-17 10:26:14 2046 TRACE nova service.wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2013-02-17 10:26:14 2046 TRACE nova return self._exit_event.wait()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-02-17 10:26:14 2046 TRACE nova return hubs.get_hub().switch()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-02-17 10:26:14 2046 TRACE nova return self.greenlet.switch()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2013-02-17 10:26:14 2046 TRACE nova result = function(*args, **kwargs)
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 132, in run_server
2013-02-17 10:26:14 2046 TRACE nova server.start()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 398, in start
2013-02-17 10:26:14 2046 TRACE nova self.manager.init_host()
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 297, in init_host
2013-02-17 10:26:14 2046 TRACE nova self.driver.plug_vifs(instance, legacy_net_info)
2013-02-17 10:26:14 2046 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 439, in plug_vifs
2013-02-17 10:26:14 2046...

Read more...

Revision history for this message
Thiago Martins (martinx) wrote :

The package (1.4.0-1ubuntu1.5~ubuntu12.04.1~ppa1) related on comment #10 (from James PPA) is working Okay.

Problem fixed with James PPA.

Thanks!
Thiago

Revision history for this message
Thiago Martins (martinx) wrote :

BTW, my Openstack environment, powered by Nova Network with Open vSwitch via brcompat module is working smoothly with openvswitch-1.4.0-1ubuntu1.5~ubuntu12.04.1~ppa1.

Cheers!
Thiago

Revision history for this message
Peng Yong (ppyy) wrote :

sorry, remove and install again, it works now

Revision history for this message
Colin Watson (cjwatson) wrote : Please test proposed package

Hello Sebastian, or anyone else affected,

Accepted openvswitch into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openvswitch/1.4.0-1ubuntu1.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in openvswitch (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in openvswitch (Ubuntu Quantal):
status: In Progress → Fix Committed
Revision history for this message
Colin Watson (cjwatson) wrote :

Hello Sebastian, or anyone else affected,

Accepted openvswitch into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openvswitch/1.4.3-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Peng Yong (ppyy)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Sebastian Marsching (sebastian-marsching) wrote :

I tested the following packages from precise-proposed on Ubuntu 12.04 LTS (amd64) and they work fine for me:
openvswitch-brcompat_1.4.0-1ubuntu1.5
openvswitch-common_1.4.0-1ubuntu1.5
openvswitch-controller_1.4.0-1ubuntu1.5
openvswitch-datapath-dkms_1.4.0-1ubuntu1.5
openvswitch-pki_1.4.0-1ubuntu1.5
openvswitch-switch_1.4.0-1ubuntu1.5

Revision history for this message
Thiago Martins (martinx) wrote : Re: [Bug 1125611] Re: DKMS brcompat module circular dependency causes broken module

Version 1.4.0-1ubuntu1.5 working fine.
Openstack Cloud Computing (Nova Network + brcompat) environment okay.

Thanks!
Thiago

On 18 February 2013 16:55, Sebastian Marsching <email address hidden>wrote:

> I tested the following packages from precise-proposed on Ubuntu 12.04 LTS
> (amd64) and they work fine for me:
> openvswitch-brcompat_1.4.0-1ubuntu1.5
> openvswitch-common_1.4.0-1ubuntu1.5
> openvswitch-controller_1.4.0-1ubuntu1.5
> openvswitch-datapath-dkms_1.4.0-1ubuntu1.5
> openvswitch-pki_1.4.0-1ubuntu1.5
> openvswitch-switch_1.4.0-1ubuntu1.5
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1125611
>
> Title:
> DKMS brcompat module circular dependency causes broken module
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1125611/+subscriptions
>

Revision history for this message
Colin Watson (cjwatson) wrote :

Waiting the usual seven-day waiting period since this is a regression.

tags: added: regression-update
Revision history for this message
Colin Watson (cjwatson) wrote :

s/Waiting/Waiving/

Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 1.4.0-1ubuntu1.5

---------------
openvswitch (1.4.0-1ubuntu1.5) precise-proposed; urgency=high

  * Fix broken brcompat module due to circular DKMS build dependency
    causing brcompat.o to be excluded from module (LP: #1125611):
    - d/p/0009-datapath-rename-brcompat.c-to-brcompat_main.c.patch:
      Cherry picked fix from upstream VCS which renames brcompat.c ->
      brcompat_main.c to support drop of _mod suffix.
 -- James Page <email address hidden> Sat, 16 Feb 2013 13:12:54 +0000

Changed in openvswitch (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 1.4.3-0ubuntu2.1

---------------
openvswitch (1.4.3-0ubuntu2.1) quantal-proposed; urgency=high

  * Fix broken brcompat module due to circular DKMS build dependency
    causing brcompat.o to be excluded from module (LP: #1125611):
    - d/p/0009-datapath-rename-brcompat.c-to-brcompat_main.c.patch:
      Cherry picked fix from upstream VCS which renames brcompat.c ->
      brcompat_main.c to support drop of _mod postfix.
  * Fix module-assistant installs of openvswitch-datapath (LP: #1088160):
    - d/rules.modules: Drop _mod postfix when installing modules.
 -- James Page <email address hidden> Sat, 16 Feb 2013 14:02:05 +0000

Changed in openvswitch (Ubuntu Quantal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.