| 2017-01-10 20:27:50 |
Mike Pontillo |
bug |
|
|
added bug |
| 2017-01-10 20:28:21 |
Mike Pontillo |
bug task added |
|
netaddr |
|
| 2017-01-10 20:28:35 |
Mike Pontillo |
bug task added |
|
maas |
|
| 2017-01-10 20:28:52 |
Mike Pontillo |
nominated for series |
|
maas/2.1 |
|
| 2017-01-10 20:28:52 |
Mike Pontillo |
bug task added |
|
maas/2.1 |
|
| 2017-01-10 20:28:52 |
Mike Pontillo |
nominated for series |
|
maas/2.2 |
|
| 2017-01-10 20:28:52 |
Mike Pontillo |
bug task added |
|
maas/2.2 |
|
| 2017-01-10 20:29:02 |
Mike Pontillo |
maas: status |
New |
Won't Fix |
|
| 2017-01-10 20:29:06 |
Mike Pontillo |
maas/2.1: status |
New |
Won't Fix |
|
| 2017-01-10 20:29:09 |
Mike Pontillo |
bug task deleted |
maas/2.2 |
|
|
| 2017-01-10 20:29:18 |
Mike Pontillo |
python-netaddr (Ubuntu): status |
New |
Confirmed |
|
| 2017-01-10 20:33:14 |
Mike Pontillo |
description |
The Debian package for `netaddr` ships with a pre-calculated index for entries in the IEEE OUI and IAB files, which the `ieee-data` package places here:
$ dpkg -L python-netaddr | grep idx$
/usr/lib/python2.7/dist-packages/netaddr/eui/oui.idx
/usr/lib/python2.7/dist-packages/netaddr/eui/iab.idx
The index files only work if they match the IEEE data at the time the `netaddr` package was built. If anything changes in the IEEE data, the seek locations in the index will be thrown off, causing undefined behavior.
However, the way `netaddr` is packaged doesn't guarantee that the index will be kept in sync with the data files in the `ieee-data` pacakge. In the `netaddr` packages, symbolic links are created to the data files in the `ieee-data` package as follows:
$ dpkg -L ieee-data | grep usr.share.*.txt
/usr/share/ieee-data/iab.txt
/usr/share/ieee-data/oui.txt
$ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt
Upstream, `netaddr` ships with its own copy of these files. During the build process, `netaddr` generates an index file for fast lookups into the OUE and IAB text files.
However, there is a problem with this arrangement that only occurs in the Debian package: if the `ieee-data` on a given system is out-of-sync with the index in the `python-netaddr` or `python3-netaddr`, the undefined behavior begins.
In MAAS, we make heavy use of netaddr, and the symptoms of the index file being out of date have ranged from an IndexError being raised to UnicodeDecodeError, since `netaddr` assumes that the index will always match the data file, and caches the index for the life of the process.[1]
Symptoms
========
This bug can cause `netaddr` to raise tracebacks during operation, such as `IndexError`, and possibly others, as a result of the incorrect indexes. For examples, see also:
Bug #1655049
Bug #1628761
Suggested ways to fix
=====================
(1) The `netaddr` package MUST always be in a state where the index file is consistent with the IEEE data.
(1a) The `netaddr` package COULD ship with a copy of the IEEE data files as-designed, so that there is never a mismatch regarding the on-disk data. (This is the simplest fix, but is likely a Debian policy violation due to the duplication of data.)
(1b) The `netaddr` package COULD make a copy of the current state of the OUI and IAB indexes in `ieee-data` when the package is installed or updated. (Perhaps hard links would work, so that `netaddr` can continue working with the old data until the indexes are regenerated. But it would need to fall back to a data copy in case of cross-device links.)
(1c) If the `ieee-data` package changes, and the `netaddr` is using the data from it, the `netaddr` package MUST correspondingly update its data files and index.
(2) The `netaddr` upstream code MUST detect if the index file changes and reload the index (to be usable in a long-running process backed by a Debian package that could be updated at any time).
Short term, the most important thing needed is a fix for (1), so that will be the focus for this bug.
Workaround
==========
Regenerate the indexes (for python 2.x and 3.x) as root by running:
sudo python -m netaddr.eui.ieee
sudo python3 -m netaddr.eui.ieee
Either or both commands may be needed, depending on which `netaddr` packages are installed.
Then, restart any long-running processes (such as MAAS) which may rely on `netaddr` for OUI lookups.
Note that even if this bug is fixed, the "long-running process" issue still exists. To fix that, (2) above needs to be resolved.
---
[1]: That means there is a related issue: long-running processes will be exposed to the same symptoms of this bug, if the `netaddr` package is upgraded while the long-running process is continues to run. Arguably, this is a bug that upstream should address by recalculating the index if the data file changes, and/or forcing an index recalculation if an integrity check fails, such as by storing a size and hash of the file. I assume it would be counter-productive to do too many integrity checks, though, because by the time you get done with them, you may as well have recalculated the entire index. (And on a slow embedded system, this might be unacceptable.) |
The Debian package for `netaddr` ships with a pre-calculated index for entries in the IEEE OUI and IAB files, which the `ieee-data` package places here:
$ dpkg -L python-netaddr | grep idx$
/usr/lib/python2.7/dist-packages/netaddr/eui/oui.idx
/usr/lib/python2.7/dist-packages/netaddr/eui/iab.idx
The index files only work if they match the IEEE data at the time the `netaddr` package was built. If anything changes in the IEEE data, the seek locations in the index will be thrown off, causing undefined behavior.
However, the way `netaddr` is packaged doesn't guarantee that the index will be kept in sync with the data files in the `ieee-data` pacakge. In the `netaddr` packages, symbolic links are created to the data files in the `ieee-data` package as follows:
$ dpkg -L ieee-data | grep usr.share.*.txt
/usr/share/ieee-data/iab.txt
/usr/share/ieee-data/oui.txt
$ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt
Upstream, `netaddr` ships with its own copy of these files. During the build process, `netaddr` generates an index file for fast lookups into the OUI and IAB text files.
However, there is a problem with this arrangement that only occurs in the Debian package: if the `ieee-data` on a given system is out-of-sync with the index in the `python-netaddr` or `python3-netaddr`, the undefined behavior begins.
In MAAS, we make heavy use of netaddr, and the symptoms of the index file being out of date have ranged from an IndexError being raised to UnicodeDecodeError, since `netaddr` assumes that the index will always match the data file, and caches the index for the life of the process.[1]
Symptoms
========
This bug can cause `netaddr` to raise tracebacks during operation, such as `IndexError`, and possibly others, as a result of the incorrect indexes. For examples, see also:
Bug #1655049
Bug #1628761
Suggested ways to fix
=====================
(1) The `netaddr` package MUST always be in a state where the index file is consistent with the IEEE data.
(1a) The `netaddr` package COULD ship with a copy of the IEEE data files as-designed, so that there is never a mismatch regarding the on-disk data. (This is the simplest fix, but is likely a Debian policy violation due to the duplication of data.)
(1b) The `netaddr` package COULD make a copy of the current state of the OUI and IAB indexes in `ieee-data` when the package is installed or updated. (Perhaps hard links would work, so that `netaddr` can continue working with the old data until the indexes are regenerated. But it would need to fall back to a data copy in case of cross-device links.)
(1c) If the `ieee-data` package changes, and the `netaddr` is using the data from it, the `netaddr` package MUST correspondingly update its data files and index.
(2) The `netaddr` upstream code MUST detect if the index file changes and reload the index (to be usable in a long-running process backed by a Debian package that could be updated at any time).
Short term, the most important thing needed is a fix for (1), so that will be the focus for this bug.
Workaround
==========
Regenerate the indexes (for python 2.x and 3.x) as root by running:
sudo python -m netaddr.eui.ieee
sudo python3 -m netaddr.eui.ieee
Either or both commands may be needed, depending on which `netaddr` packages are installed.
Then, restart any long-running processes (such as MAAS) which may rely on `netaddr` for OUI lookups.
Note that even if this bug is fixed, the "long-running process" issue still exists. To fix that, (2) above needs to be resolved.
---
[1]: That means there is a related issue: long-running processes will be exposed to the same symptoms of this bug, if the `netaddr` package is upgraded while the long-running process is continues to run. Arguably, this is a bug that upstream should address by recalculating the index if the data file changes, and/or forcing an index recalculation if an integrity check fails, such as by storing a size and hash of the file. I assume it would be counter-productive to do too many integrity checks, though, because by the time you get done with them, you may as well have recalculated the entire index. (And on a slow embedded system, this might be unacceptable.) |
|
| 2017-01-10 20:36:07 |
Mike Pontillo |
maas: importance |
Undecided |
Critical |
|
| 2017-01-10 20:36:10 |
Mike Pontillo |
maas/2.1: importance |
Undecided |
Critical |
|
| 2017-01-10 21:19:28 |
Mike Pontillo |
description |
The Debian package for `netaddr` ships with a pre-calculated index for entries in the IEEE OUI and IAB files, which the `ieee-data` package places here:
$ dpkg -L python-netaddr | grep idx$
/usr/lib/python2.7/dist-packages/netaddr/eui/oui.idx
/usr/lib/python2.7/dist-packages/netaddr/eui/iab.idx
The index files only work if they match the IEEE data at the time the `netaddr` package was built. If anything changes in the IEEE data, the seek locations in the index will be thrown off, causing undefined behavior.
However, the way `netaddr` is packaged doesn't guarantee that the index will be kept in sync with the data files in the `ieee-data` pacakge. In the `netaddr` packages, symbolic links are created to the data files in the `ieee-data` package as follows:
$ dpkg -L ieee-data | grep usr.share.*.txt
/usr/share/ieee-data/iab.txt
/usr/share/ieee-data/oui.txt
$ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt
Upstream, `netaddr` ships with its own copy of these files. During the build process, `netaddr` generates an index file for fast lookups into the OUI and IAB text files.
However, there is a problem with this arrangement that only occurs in the Debian package: if the `ieee-data` on a given system is out-of-sync with the index in the `python-netaddr` or `python3-netaddr`, the undefined behavior begins.
In MAAS, we make heavy use of netaddr, and the symptoms of the index file being out of date have ranged from an IndexError being raised to UnicodeDecodeError, since `netaddr` assumes that the index will always match the data file, and caches the index for the life of the process.[1]
Symptoms
========
This bug can cause `netaddr` to raise tracebacks during operation, such as `IndexError`, and possibly others, as a result of the incorrect indexes. For examples, see also:
Bug #1655049
Bug #1628761
Suggested ways to fix
=====================
(1) The `netaddr` package MUST always be in a state where the index file is consistent with the IEEE data.
(1a) The `netaddr` package COULD ship with a copy of the IEEE data files as-designed, so that there is never a mismatch regarding the on-disk data. (This is the simplest fix, but is likely a Debian policy violation due to the duplication of data.)
(1b) The `netaddr` package COULD make a copy of the current state of the OUI and IAB indexes in `ieee-data` when the package is installed or updated. (Perhaps hard links would work, so that `netaddr` can continue working with the old data until the indexes are regenerated. But it would need to fall back to a data copy in case of cross-device links.)
(1c) If the `ieee-data` package changes, and the `netaddr` is using the data from it, the `netaddr` package MUST correspondingly update its data files and index.
(2) The `netaddr` upstream code MUST detect if the index file changes and reload the index (to be usable in a long-running process backed by a Debian package that could be updated at any time).
Short term, the most important thing needed is a fix for (1), so that will be the focus for this bug.
Workaround
==========
Regenerate the indexes (for python 2.x and 3.x) as root by running:
sudo python -m netaddr.eui.ieee
sudo python3 -m netaddr.eui.ieee
Either or both commands may be needed, depending on which `netaddr` packages are installed.
Then, restart any long-running processes (such as MAAS) which may rely on `netaddr` for OUI lookups.
Note that even if this bug is fixed, the "long-running process" issue still exists. To fix that, (2) above needs to be resolved.
---
[1]: That means there is a related issue: long-running processes will be exposed to the same symptoms of this bug, if the `netaddr` package is upgraded while the long-running process is continues to run. Arguably, this is a bug that upstream should address by recalculating the index if the data file changes, and/or forcing an index recalculation if an integrity check fails, such as by storing a size and hash of the file. I assume it would be counter-productive to do too many integrity checks, though, because by the time you get done with them, you may as well have recalculated the entire index. (And on a slow embedded system, this might be unacceptable.) |
Before reading further, note that this bug report is for the bug in the Debian package for `netaddr`. Bug #1655452 is for the bug in upstream `netaddr` code (related to cached index coherency).
---
The Debian package for `netaddr` ships with a pre-calculated index for entries in the IEEE OUI and IAB files, which the `ieee-data` package places here:
$ dpkg -L python-netaddr | grep idx$
/usr/lib/python2.7/dist-packages/netaddr/eui/oui.idx
/usr/lib/python2.7/dist-packages/netaddr/eui/iab.idx
The index files only work if they match the IEEE data at the time the `netaddr` package was built. If anything changes in the IEEE data, the seek locations in the index will be thrown off, causing undefined behavior.
However, the way `netaddr` is packaged doesn't guarantee that the index will be kept in sync with the data files in the `ieee-data` pacakge. In the `netaddr` packages, symbolic links are created to the data files in the `ieee-data` package as follows:
$ dpkg -L ieee-data | grep usr.share.*.txt
/usr/share/ieee-data/iab.txt
/usr/share/ieee-data/oui.txt
$ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt
Upstream, `netaddr` ships with its own copy of these files. During the build process, `netaddr` generates an index file for fast lookups into the OUI and IAB text files.
However, there is a problem with this arrangement that only occurs in the Debian package: if the `ieee-data` on a given system is out-of-sync with the index in the `python-netaddr` or `python3-netaddr`, the undefined behavior begins.
In MAAS, we make heavy use of netaddr, and the symptoms of the index file being out of date have ranged from an IndexError being raised to UnicodeDecodeError, since `netaddr` assumes that the index will always match the data file, and caches the index for the life of the process.[1]
Symptoms
========
This bug can cause `netaddr` to raise tracebacks during operation, such as `IndexError`, and possibly others, as a result of the incorrect indexes. For examples, see also:
Bug #1655049
Bug #1628761
Suggested ways to fix
=====================
(1) The `netaddr` package MUST always be in a state where the index file is consistent with the IEEE data.
(1a) The `netaddr` package COULD ship with a copy of the IEEE data files as-designed, so that there is never a mismatch regarding the on-disk data. (This is the simplest fix, but is likely a Debian policy violation due to the duplication of data.)
(1b) The `netaddr` package COULD make a copy of the current state of the OUI and IAB indexes in `ieee-data` when the package is installed or updated. (Perhaps hard links would work, so that `netaddr` can continue working with the old data until the indexes are regenerated. But it would need to fall back to a data copy in case of cross-device links.)
(1c) If the `ieee-data` package changes, and the `netaddr` is using the data from it, the `netaddr` package MUST correspondingly update its data files and index.
(2) The `netaddr` upstream code MUST detect if the index file changes and reload the index (to be usable in a long-running process backed by a Debian package that could be updated at any time).
Short term, the most important thing needed is a fix for (1), so that will be the focus for this bug.
Workaround
==========
Regenerate the indexes (for python 2.x and 3.x) as root by running:
sudo python -m netaddr.eui.ieee
sudo python3 -m netaddr.eui.ieee
Either or both commands may be needed, depending on which `netaddr` packages are installed.
Then, restart any long-running processes (such as MAAS) which may rely on `netaddr` for OUI lookups.
Note that even if this bug is fixed, the "long-running process" issue still exists. To fix that, (2) above needs to be resolved.
---
[1]: That means there is a related issue: long-running processes will be exposed to the same symptoms of this bug, if the `netaddr` package is upgraded while the long-running process is continues to run. Arguably, this is a bug that upstream should address by recalculating the index if the data file changes, and/or forcing an index recalculation if an integrity check fails, such as by storing a size and hash of the file. I assume it would be counter-productive to do too many integrity checks, though, because by the time you get done with them, you may as well have recalculated the entire index. (And on a slow embedded system, this might be unacceptable.) |
|
| 2017-01-10 21:19:39 |
Mike Pontillo |
bug task deleted |
netaddr |
|
|
| 2018-02-07 23:45:02 |
Mike Pontillo |
maas: status |
Won't Fix |
Triaged |
|
| 2018-02-07 23:45:03 |
Mike Pontillo |
maas/2.1: status |
Won't Fix |
Triaged |
|
| 2018-02-07 23:45:06 |
Mike Pontillo |
maas: assignee |
|
Mike Pontillo (mpontillo) |
|
| 2018-02-07 23:45:10 |
Mike Pontillo |
bug task deleted |
maas/2.1 |
|
|
| 2018-02-07 23:45:43 |
Mike Pontillo |
nominated for series |
|
maas/2.3 |
|
| 2018-02-07 23:45:43 |
Mike Pontillo |
bug task added |
|
maas/2.3 |
|
| 2018-02-07 23:45:49 |
Mike Pontillo |
maas/2.3: status |
New |
Triaged |
|
| 2018-02-07 23:45:51 |
Mike Pontillo |
maas/2.3: importance |
Undecided |
Critical |
|
| 2018-02-07 23:45:53 |
Mike Pontillo |
maas/2.3: assignee |
|
Mike Pontillo (mpontillo) |
|
| 2018-02-07 23:45:56 |
Mike Pontillo |
maas: milestone |
|
2.4.0alpha1 |
|
| 2018-02-07 23:45:58 |
Mike Pontillo |
maas/2.3: milestone |
|
2.3.x |
|
| 2018-02-14 18:54:49 |
Andres Rodriguez |
maas: milestone |
2.4.0alpha1 |
2.4.0alpha2 |
|
| 2018-03-13 12:20:49 |
Andres Rodriguez |
maas: milestone |
2.4.0alpha2 |
2.4.0beta1 |
|
| 2018-03-16 17:42:43 |
Andres Rodriguez |
maas: importance |
Critical |
High |
|
| 2018-03-16 17:42:45 |
Andres Rodriguez |
maas/2.3: importance |
Critical |
High |
|
| 2018-03-22 23:25:55 |
Andres Rodriguez |
maas: milestone |
2.4.0beta1 |
2.4.0beta2 |
|
| 2018-04-04 23:05:52 |
Andres Rodriguez |
maas: milestone |
2.4.0beta2 |
2.4.0beta3 |
|
| 2018-05-03 07:42:03 |
Andres Rodriguez |
maas: milestone |
2.4.0beta3 |
2.4.0beta4 |
|
| 2018-05-17 15:04:54 |
Andres Rodriguez |
maas: milestone |
2.4.0beta4 |
2.4.x |
|
| 2018-08-17 14:42:45 |
Andres Rodriguez |
maas/2.3: milestone |
2.3.x |
2.3.6 |
|
| 2018-09-05 21:46:28 |
Andres Rodriguez |
maas: milestone |
2.4.x |
2.5.x |
|
| 2019-05-31 14:11:21 |
Andres Rodriguez |
maas/2.3: milestone |
2.3.6 |
2.3.x |
|
| 2022-03-17 09:45:31 |
Jerzy Husakowski |
bug task deleted |
maas/2.3 |
|
|
| 2022-03-17 09:45:42 |
Jerzy Husakowski |
maas: status |
Triaged |
Won't Fix |
|
