Functional gate job broken due to os-log-merger crashing on null-bytes

Bug #1672921 reported by Manjeet Singh Bhatia
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Miguel Angel Ajo
os-log-merger
Fix Released
Critical
Ihar Hrachyshka
Revision history for this message
Manjeet Singh Bhatia (manjeet-s-bhatia) wrote :

oslomerger looks like culprit

017-03-14 21:05:35.372834 | 2017-03-14 21:05:35.372 | sys.exit(main())
2017-03-14 21:05:35.373974 | 2017-03-14 21:05:35.373 | File "/tmp/os-log-merger/local/lib/python2.7/site-packages/oslogmerger/oslogmerger.py", line 648, in main
2017-03-14 21:05:35.375069 | 2017-03-14 21:05:35.374 | process_logs(cfg)
2017-03-14 21:05:35.376277 | 2017-03-14 21:05:35.375 | File "/tmp/os-log-merger/local/lib/python2.7/site-packages/oslogmerger/oslogmerger.py", line 382, in process_logs
2017-03-14 21:05:35.377377 | 2017-03-14 21:05:35.377 | for entry in method(logs):
2017-03-14 21:05:35.378502 | 2017-03-14 21:05:35.378 | File "/tmp/os-log-merger/local/lib/python2.7/site-packages/oslogmerger/oslogmerger.py", line 349, in process_logs_memory_hog
2017-03-14 21:05:35.379766 | 2017-03-14 21:05:35.379 | for entry in log:
2017-03-14 21:05:35.380930 | 2017-03-14 21:05:35.380 | File "/tmp/os-log-merger/local/lib/python2.7/site-packages/oslogmerger/oslogmerger.py", line 230, in next
2017-03-14 21:05:35.382000 | 2017-03-14 21:05:35.381 | self.entry, self.next_entry = self._readline(self.next_entry)
2017-03-14 21:05:35.383102 | 2017-03-14 21:05:35.382 | File "/tmp/os-log-merger/local/lib/python2.7/site-packages/oslogmerger/oslogmerger.py", line 213, in _readline
2017-03-14 21:05:35.384124 | 2017-03-14 21:05:35.383 | **self.entry_kwargs)
2017-03-14 21:05:35.385297 | 2017-03-14 21:05:35.385 | File "/tmp/os-log-merger/local/lib/python2.7/site-packages/oslogmerger/oslogmerger.py", line 129, in factory
2017-03-14 21:05:35.386391 | 2017-03-14 21:05:35.386 | self.date = self.parse_date(prepared_line)
2017-03-14 21:05:35.387540 | 2017-03-14 21:05:35.387 | File "/tmp/os-log-merger/local/lib/python2.7/site-packages/oslogmerger/oslogmerger.py", line 97, in parse_date
2017-03-14 21:05:35.388630 | 2017-03-14 21:05:35.388 | dt = datetime.strptime(line, self.date_format)
2017-03-14 21:05:35.389844 | 2017-03-14 21:05:35.389 | TypeError: strptime() argument 1 must be string without null bytes, not str

tags: added: gate-failure
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :
summary: - Functional gate job seems broken
+ Functional gate job broken due to os-log-merger crashing on null-bytes
Changed in neutron:
importance: Undecided → Critical
status: New → In Progress
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
tags: added: functional-tests
Changed in os-log-merger:
status: New → Fix Committed
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/445884

Changed in neutron:
assignee: Ihar Hrachyshka (ihar-hrachyshka) → Miguel Angel Ajo (mangelajo)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/445884
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=75612f200b984c9f42558ed951ae19e17c3633d9
Submitter: Jenkins
Branch: master

commit 75612f200b984c9f42558ed951ae19e17c3633d9
Author: Miguel Angel Ajo <email address hidden>
Date: Wed Mar 15 12:29:32 2017 +0100

    Bump os-log-merger version in post_test_hook

    The 1.0.6 version of os-log-merger exploded in cases where
    log files contained \0 characters. 1.1.0 fixes such issue.

    Change-Id: I470d38dfd5dd7e9cf6317d9f2f32875f33f3a996
    Closes-Bug: #1672921

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/445706
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9c7ca5cc020a93a65fc817ae960d221d43e0e924
Submitter: Jenkins
Branch: master

commit 9c7ca5cc020a93a65fc817ae960d221d43e0e924
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Mar 8 05:30:04 2017 +0000

    Don't return null-byte separated string from ExternalProcess.cmdline()

    Instead construct the string using psutil.Process.cmdline() method that
    will parse the data from kernel for us and will return a proper
    space-separated command line (plus will provide some more safety
    guarantees, e.g. against zombies).

    This change also introduces a new psutil dependency for neutron. Before
    that, the library was used in functional tests only, and with a minimal
    version number that is not compatible with global-requirements.txt or
    .cmdline(). Moved the requirement into runtime requirements.

    Related-Bug: #1672921
    Change-Id: I780ea10bd1870c86c1a048071863843bc2993958

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.0.0b1

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.

Changed in os-log-merger:
status: Fix Committed → Fix Released
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.