OEM Priority Project

the summary() in bin/pm_test.py missing some test cases condition when QA runs a bigger test plan

Bug #1954375 reported by jeremyszu
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Checkbox Provider - Base
Expired
High
jeremyszu
OEM Priority Project
New
Undecided
Unassigned

Bug Description

When a checkbox session contains many pm_test calls[1], then it will append the "Completed S3 cycle(s) \n" and "Completed s2idle cycle(s) \n" to the same path (e.g. --log=/var/tmp/checkbox-ng/sessions/session_title-2021-12-06T10.25.57.session/session-share/fwts).

When the second (or later) pm_test be called, it will count previous fwts logs.

```
                with open(fwts_log_path, 'rt') as f:
                    magic_line_s3 = 'Completed S3 cycle(s) \n'
                    magic_line_s2idle = 'Completed s2idle cycle(s) \n'
                    lines = f.readlines()
                    count_s3 = lines.count(magic_line_s3)
                    count_s2idle = lines.count(magic_line_s2idle)
                    count = count_s3 + count_s2idle
                    if count != total_suspends_expected:
                        problems = (
                            "Found {} occurrences of S3/S2idle."
                            " Expected {}".format(
                                count, total_suspends_expected))
```

In the case that Stella met, it shows:
```
[Actual result]
Stress_suspend_30_cycles_with_reboots test failed
Log:Found 126 occurrences of S3/S2idle. Expected 90
```

The potential solution is to create fwts-cmd.log per test case or check the timestamp[2] whether inside the range.

Feel free to find the detail logs on https://bugs.launchpad.net/stella/+bug/1953597.

[1] https://bugs.launchpad.net/stella/+bug/1953597
[2] "This test run on 06/12/21 at 22:21:08 on host Linux u 5.14.0-1007-oem #7-Ubuntu"

See original description
jeremyszu (os369510)
tags: added: oem-priority originate-from-1953597 stella
Revision history for this message
jeremyszu (os369510) wrote :
Download full text (28.5 KiB)

Copied the comment from private bug:

```
There is no log, the log been cut.
```
1:================================= Iteration 1 ==================================
9:This test run on 06/12/21 at 05:36:14 on host Linux u 5.14.0-1007-oem #7-Ubuntu
35:Completed S3 cycle(s)
72:================================= Iteration 2 ==================================
80:This test run on 06/12/21 at 05:37:08 on host Linux u 5.14.0-1007-oem #7-Ubuntu
106:Completed S3 cycle(s)
143:================================= Iteration 3 ==================================
151:This test run on 06/12/21 at 05:38:02 on host Linux u 5.14.0-1007-oem #7-Ubuntu
177:Completed S3 cycle(s)
214:================================= Iteration 4 ==================================
222:This test run on 06/12/21 at 05:38:56 on host Linux u 5.14.0-1007-oem #7-Ubuntu
...
```
Dec 06 06:22:13 u pm_test.py.desktop[1529]: [Desktop Entry]
Dec 06 06:22:13 u pm_test.py.desktop[1529]: Name=reboot test
Dec 06 06:22:13 u pm_test.py.desktop[1529]: Comment=Verify reboot works properly
Dec 06 06:22:13 u pm_test.py.desktop[1529]: Exec=sudo /usr/bin/python3 /usr/lib/plainbox-provider-checkbox/bin/pm_test.py -r 0 -w 0 --hardware-delay 30 --pm-delay 5 --min-pm-time 0 --max-pm-time 300 --append --to
Dec 06 06:22:13 u pm_test.py.desktop[1529]: Type=Application
Dec 06 06:22:13 u pm_test.py.desktop[1529]: X-GNOME-Autostart-enabled=true
Dec 06 06:22:13 u pm_test.py.desktop[1529]: Hidden=false
Dec 06 06:22:13 u pm_test.py.desktop[1529]: INFO reboot operations remaining: 1
Dec 06 06:22:13 u pm_test.py.desktop[1529]: INFO Reboot time: 0:27:46.401842
Dec 06 06:22:13 u pm_test.py.desktop[1529]: INFO Running suspend/resume cycles
Dec 06 06:22:13 u pm_test.py.desktop[1529]: DEBUG Executing: 'checkbox-support-fwts_test --log=/var/tmp/checkbox-ng/sessions/session_title-2021-12-06T10.25.57.session/session-share/fwts -s s3 --s3-device-check
```
241:================================= Iteration 1 ==================================
249:This test run on 06/12/21 at 05:54:27 on host Linux u 5.14.0-1007-oem #7-Ubuntu
277:Completed S3 cycle(s)
315:================================= Iteration 2 ==================================
323:This test run on 06/12/21 at 05:55:03 on host Linux u 5.14.0-1007-oem #7-Ubuntu
349:Completed S3 cycle(s)
386:================================= Iteration 3 ==================================
394:This test run on 06/12/21 at 05:55:57 on host Linux u 5.14.0-1007-oem #7-Ubuntu
420:Completed S3 cycle(s)
457:================================= Iteration 4 ==================================
465:This test run on 06/12/21 at 05:56:51 on host Linux u 5.14.0-1007-oem #7-Ubuntu
491:Completed S3 cycle(s)
528:================================= Iteration 5 ==================================
536:This test run on 06/12/21 at 05:57:45 on host Linux u 5.14.0-1007-oem #7-Ubuntu
562:Completed S3 cycle(s)
599:================================= Iteration 6 ==================================
607:This test run on 06/12/21 at 05:58:39 on host Linux u 5.14.0-1007-oem #7-Ubuntu
633:Completed S3 cycle(s)
670:================================= Iteration 7 ==================================
678:This test run on 06/12/21 at 05:59:33 on host L...

description: updated
jeremyszu (os369510)
Changed in plainbox-provider-checkbox:
assignee: nobody → jeremyszu (os369510)
importance: Undecided → High
status: New → Triaged
status: Triaged → Confirmed
Matias Piipari (mz2)
tags: added: cbox-16
Revision history for this message
Maksim Beliaev (beliaev-maksim) wrote :

Bug was migrated to GitHub: https://github.com/canonical/checkbox/issues/89.
Bug is no more monitored here.

Changed in plainbox-provider-checkbox:
status: Confirmed → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.