Ubuntu
linux package

openvswitch: failed to flow_del (No such file or directory)

Bug #1408972 reported by James Page
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Invalid
High
Joseph Salisbury
Trusty
Fix Released
High
Joseph Salisbury
openvswitch (Ubuntu)
Invalid
Undecided
Unassigned
Trusty
New
Undecided
Unassigned

Bug Description

As part of the investigation into bug 1336555, we've noticed a large number of these error messages being logged in 14.04 OpenStack deployments:

2015-01-09T10:50:53.643Z|03976|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)
2015-01-09T10:50:54.644Z|03977|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)
2015-01-09T10:50:55.645Z|03978|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)
2015-01-09T10:50:56.645Z|03979|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)
2015-01-09T10:50:57.646Z|03980|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)
2015-01-09T10:50:58.646Z|03981|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)
2015-01-09T10:50:59.645Z|03982|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)
2015-01-09T10:51:00.646Z|03983|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)
2015-01-09T10:51:01.646Z|03984|dpif|WARN|system@ovs-system: failed to flow_del (No such file or directory) skb_priority(0),in_port(4),skb_mark(0),eth(src=ce:2b:3a:c3:d7:d1,dst=33:33:ff:50:c2:61),eth_type(0x86dd),ipv6(src=::,dst=ff02::1:ff50:c261,label=0,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=fe80::5ccd:caff:fe50:c261)

this was fixed upstream in the datapath dkms module (see https://github.com/openvswitch/ovs/commit/3601bd879) and is fixed in the 3.16 kernel as well; please can this fix be backported to 3.13.

Thanks!

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-generic 3.13.0.43.50
ProcVersionSignature: Ubuntu 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Dec 11 21:01 seq
 crw-rw---- 1 root audio 116, 33 Dec 11 21:01 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Fri Jan 9 10:50:39 2015
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL360p Gen8
PciMultimedia:

ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-35-generic root=UUID=f8e77ea1-c970-4e7a-9e49-48532d763167 ro console=tty0 console=ttyS1,38400 quiet
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-35-generic N/A
 linux-backports-modules-3.13.0-35-generic N/A
 linux-firmware 1.127.10
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/21/2012
dmi.bios.vendor: HP
dmi.bios.version: P71
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP71:bd02/21/2012:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL360p Gen8
dmi.sys.vendor: HP

CVE References

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Raising as task for openvswitch as well as we should fix the datapath module in the 14.04 packaging - this gets backported to 12.04 - users here will be impacted on older kernel versions where the use of the dkms module is required.

Revision history for this message
Brad Figg (brad-figg) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
James Page (james-page) wrote :

Fix already present in datapath module for 2.0.2.

Changed in openvswitch (Ubuntu):
status: New → Invalid
Ryan Beisner (1chb1n)
tags: added: openstack uosci
Joseph Salisbury (jsalisbury)
tags: added: kernel-da-key
Changed in linux (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Looks like this is the commit that fixes the issue in the 3.16 kernel:

commit 4a46b24e147dfa9b858026da02cad0bdd4e149d2
Author: Alex Wang <email address hidden>
Date: Mon Jun 30 20:30:29 2014 -0700

    openvswitch: Use exact lookup for flow_get and flow_del.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Commit 4a46b24e1 does not cherry-pick cleanly in Trusty, so some other commits may be needed.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

There seems to be a few commits that are needed to cherry-pick 4a46b24 cleanly, or a considerable backport is needed.

Is using the Utopic backport kernel for Trusty(14.04.2) an option?

Revision history for this message
James Page (james-page) wrote :

I'd prefer that we have a good story with 3.13; alot of deployments probably won't use the HWE kernels due to the shorter support lifecycle.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

I created a backport of commit 4a46b24 and built a test kernel. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1408972/

Can you test this kernel and see if it resolves this bug?

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Also, you need to install the linux-image and linux-image-extra .deb packages.

Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
importance: Medium → High
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

@James

Just curious if you had a chance to test the kernel posted in comment #9?

Revision history for this message
James Page (james-page) wrote :

Joseph - not just yet - we have a charm release this week - hopefully we can deploy your test kernel on Friday.

Revision history for this message
James Page (james-page) wrote :

Joseph

I've patched your test kernel into one of our cloud nodes - lets see how it goes :-)

Revision history for this message
James Page (james-page) wrote :
Download full text (3.5 KiB)

The warning messages from ovs have disappeared:

2015-02-24T06:59:27.667Z|01948|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2015-02-24T07:21:26.499Z|01949|bridge|INFO|bridge br-int: added interface qvoae80d4db-ed on port 317
2015-02-24T07:21:37.536Z|01950|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)
2015-02-24T07:35:07.440Z|01951|bridge|INFO|bridge br-int: added interface qvof7aa16b4-e2 on port 318
2015-02-24T07:35:20.813Z|01952|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)
2015-02-24T07:36:16.509Z|01953|bridge|INFO|bridge br-int: added interface qvo728d4925-f9 on port 319
2015-02-24T07:36:28.125Z|01954|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)
2015-02-24T07:37:07.278Z|01955|bridge|INFO|bridge br-int: added interface qvo2e643a8e-e0 on port 320
2015-02-24T07:37:28.073Z|01956|ofproto|INFO|br-int: 1 flow_mods 20 s ago (1 deletes)
2015-02-24T07:38:16.861Z|01957|bridge|INFO|bridge br-int: added interface qvo4069b2ce-6f on port 321
2015-02-24T07:38:28.359Z|01958|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)
2015-02-24T07:39:26.142Z|01959|bridge|INFO|bridge br-int: added interface qvocab2db31-b0 on port 322
2015-02-24T07:39:38.362Z|01960|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)
2015-02-24T08:02:27.466Z|01961|bridge|INFO|bridge br-int: added interface qvo3a35ec04-8d on port 323
2015-02-24T08:02:29.516Z|01962|bridge|INFO|bridge br-int: added interface qvocbe0764b-7a on port 324
2015-02-24T08:02:39.184Z|01963|ofproto|INFO|br-int: 2 flow_mods in the 1 s starting 10 s ago (2 deletes)
2015-02-24T09:11:29.499Z|01964|bridge|INFO|bridge br-int: added interface qvo6109c03e-d7 on port 325
2015-02-24T09:11:42.086Z|01965|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)
2015-02-24T09:11:44.390Z|01966|bridge|INFO|bridge br-int: added interface qvo8fea2619-af on port 326
2015-02-24T09:11:59.286Z|01967|bridge|INFO|bridge br-int: added interface qvo406e935a-8e on port 327
2015-02-24T09:12:15.035Z|01968|bridge|INFO|bridge br-int: added interface qvoec8b6516-46 on port 328
2015-02-24T09:12:33.683Z|01969|bridge|INFO|bridge br-int: added interface qvo004892ce-34 on port 329
2015-02-24T09:12:41.754Z|01970|bridge|INFO|bridge br-int: added interface qvofea96788-84 on port 330
2015-02-24T09:12:41.941Z|01971|ofproto|INFO|br-int: 4 flow_mods in the 49 s starting 55 s ago (4 deletes)
2015-02-24T09:13:42.116Z|01972|ofproto|INFO|br-int: 1 flow_mods 58 s ago (1 deletes)
2015-02-24T09:38:32.682Z|01973|bridge|INFO|bridge br-int: added interface qvo1d6c44e8-d1 on port 331
2015-02-24T09:38:45.441Z|01974|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)
2015-02-24T09:38:45.441Z|01975|ofproto|INFO|br-tun: 2 flow_mods 10 s ago (1 adds, 1 modifications)
2015-02-24T09:39:03.039Z|01976|bridge|INFO|bridge br-int: added interface qvo4894ddc5-4f on port 332
2015-02-24T09:39:35.323Z|01977|bridge|INFO|bridge br-int: added interface qvo99244c03-16 on port 333
2015-02-24T09:39:45.247Z|01978|ofproto|INFO|br-int: 2 flow_mods in the 32 s starting 40 s ago (2 deletes)
2015-02-24T09:40:05.694Z|01979|bridge|INFO|bridge br-int: added interface qvo87d42fbd-c3 on port 334
2015-02-24T09:40:19.309Z|01980|bridge|INFO|bridge br-int:...

Read more...

Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
status: Confirmed → In Progress
assignee: nobody → Joseph Salisbury (jsalisbury)
Brad Figg (brad-figg)
Changed in linux (Ubuntu Trusty):
status: New → Fix Committed
Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Trusty):
importance: Undecided → High
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-trusty
James Page (james-page)
tags: added: verification-done-trusty
removed: verification-needed-trusty
Revision history for this message
James Page (james-page) wrote :

To confirm - runnig this on one of our impacted compute nodes - no error messages from ovs so +1

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (8.9 KiB)

This bug was fixed in the package linux - 3.13.0-48.80

---------------
linux (3.13.0-48.80) trusty; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1431263
  * Merged back all changes that were in Ubuntu-3.13.0-47.78

  [ Upstream Kernel Changes ]

  * xfs: remote attribute overwrite causes transaction overrun
    - LP: #1429821
    - CVE-2015-0274

linux (3.13.0-47.78) trusty; urgency=low

  [ Seth Forshee ]

  * Release Tracking Bug
    - LP: #1427733

  [ Rodrigo Vivi ]

  * SAUCE: drm/i915: Fix and clean BDW PCH identification
    - LP: #1423292
  * SAUCE: drm/i915: BDW Fix Halo PCI IDs marked as ULT.
    - LP: #1423292

  [ Upstream Kernel Changes ]

  * ext4: prevent bugon on race between write/fcntl
  * Bluetooth: ath3k: workaround the compatibility issue with xHCI
    controller
    - LP: #1400215
  * openvswitch: Silence RCU lockdep checks from flow lookup.
    - LP: #1408972
  * openvswitch: Use exact lookup for flow_get and flow_del.
    - LP: #1408972
  * splice: Apply generic position and size checks to each write
    - LP: #1416498
    - CVE-2014-7822
  * ALSA: hda - enable mute led quirk for one more hp machine.
    - LP: #1410704
  * crypto: prefix module autoloading with "crypto-"
    - LP: #1427438
  * crypto: add missing crypto module aliases
    - LP: #1427438
  * crypto: include crypto- module prefix in template
    - LP: #1427438
  * crypto: crc32c - add missing crypto module alias
    - LP: #1427438
  * drm/i915: Invalidate media caches on gen7
    - LP: #1427438
  * drm/i915: Force the CS stall for invalidate flushes
    - LP: #1427438
  * audit: restore AUDIT_LOGINUID unset ABI
    - LP: #1427438
  * parisc: fix out-of-register compiler error in ldcw inline assembler
    function
    - LP: #1427438
  * kvm: x86: drop severity of "generation wraparound" message
    - LP: #1427438
  * udf: Verify i_size when loading inode
    - LP: #1427438
  * udf: Verify symlink size before loading it
    - LP: #1427438
  * udf: Check path length when reading symlink
    - LP: #1427438
  * udf: Check component length before reading it
    - LP: #1427438
  * crypto: af_alg - fix backlog handling
    - LP: #1427438
  * ASoC: dwc: Ensure FIFOs are flushed to prevent channel swap
    - LP: #1427438
  * video/logo: prevent use of logos after they have been freed
    - LP: #1427438
  * video/fbdev: fix defio's fsync
    - LP: #1427438
  * Add USB_EHCI_EXYNOS to multi_v7_defconfig
    - LP: #1427438
  * drm/i915: Swap primary planes on gen2 for FBC
    - LP: #1427438
  * drm/i915: Don't swap planes on 830M
    - LP: #1427438
  * drm/i915: Don't call intel_prepare_page_flip() multiple times on gen2-4
    - LP: #1427438
  * x86, vdso: Use asm volatile in __getcpu
    - LP: #1427438
  * drivers: net: cpsw: enable interrupts after napi enable and clearing
    previous interrupts
    - LP: #1427438
  * net: ethernet: cpsw: fix hangs with interrupts
    - LP: #1427438
  * ALSA: hda - Fix wrong gpio_dir & gpio_mask hint setups for IDT/STAC
    codecs
    - LP: #1427438
  * drm/radeon: KV has three PPLLs (v2)
    - LP: #1427438
  * drm/radeon: properly filter DP1.2 4k modes on non-DP1.2 hw
    - LP: #1427438
  * virtio_pci: d...

Read more...

Changed in linux (Ubuntu Trusty):
status: Fix Committed → Fix Released
Mathew Hodson (mhodson)
Changed in linux (Ubuntu):
status: Fix Committed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.