ubuntu-kernel-tests

aws test from ubuntu_k8s_unit_tests failed on G-gcp-5.8

Bug #1908755 reported by Kelsey Steele on 2020-12-18

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	ubuntu-kernel-tests	New	Undecided	Unassigned

Bug Description

Only found on a couple nodes: g1-small and n1-standard-8

12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:34:11 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| 12/16 14:40:33 DEBUG| utils:0153| [stdout] E1216 14:34:11.259245 34615 aws.go:4259] Error deregistering load balancer from instance security groups: "Could not determine security group for load balancer: "
utils:0153| [stdout] --- FAIL: TestCreateDisk (0.00s)
utils:0153| [stdout] panic:
utils:0153| [stdout]
utils:0153| [stdout] mock: Unexpected Method Call
utils:0153| [stdout] -----------------------------
utils:0153| [stdout]
utils:0153| [stdout] CreateVolume(*ec2.CreateVolumeInput)
utils:0153| [stdout] 0: {
utils:0153| [stdout] AvailabilityZone: "us-east-1a",
utils:0153| [stdout] Encrypted: false,
utils:0153| [stdout] Size: 10,
utils:0153| [stdout] TagSpecifications: [{
utils:0153| [stdout] ResourceType: "volume",
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] Value: "owned"
utils:0153| [stdout] },{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] }]
utils:0153| [stdout] }],
utils:0153| [stdout] VolumeType: "gp2"
utils:0153| [stdout] }
utils:0153| [stdout]
utils:0153| [stdout] The closest call I have is:
utils:0153| [stdout]
utils:0153| [stdout] CreateVolume(*ec2.CreateVolumeInput)
utils:0153| [stdout] 0: {
utils:0153| [stdout] AvailabilityZone: "us-east-1a",
utils:0153| [stdout] Encrypted: false,
utils:0153| [stdout] Size: 10,
utils:0153| [stdout] TagSpecifications: [{
utils:0153| [stdout] ResourceType: "volume",
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] },{
utils:0153| [stdout] Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] Value: "owned"
utils:0153| [stdout] }]
utils:0153| [stdout] }],
utils:0153| [stdout] VolumeType: "gp2"
utils:0153| [stdout] }
utils:0153| [stdout]
utils:0153| [stdout] Difference found in argument 0:
utils:0153| [stdout]
utils:0153| [stdout] --- Expected
utils:0153| [stdout] +++ Actual
utils:0153| [stdout] @@ -7,7 +7,7 @@
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] + Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] + Value: "owned"
utils:0153| [stdout] + },{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] - },{
utils:0153| [stdout] - Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] - Value: "owned"
utils:0153| [stdout] }]
utils:0153| [stdout]
utils:0153| [stdout] Diff: 0: FAIL: {
utils:0153| [stdout] AvailabilityZone: "us-east-1a",
utils:0153| [stdout] Encrypted: false,
utils:0153| [stdout] Size: 10,
utils:0153| [stdout] TagSpecifications: [{
utils:0153| [stdout] ResourceType: "volume",
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] Value: "owned"
utils:0153| [stdout] },{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] }]
utils:0153| [stdout] }],
utils:0153| [stdout] VolumeType: "gp2"
utils:0153| [stdout] } != {
utils:0153| [stdout] AvailabilityZone: "us-east-1a",
utils:0153| [stdout] Encrypted: false,
utils:0153| [stdout] Size: 10,
utils:0153| [stdout] TagSpecifications: [{
utils:0153| [stdout] ResourceType: "volume",
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] },{
utils:0153| [stdout] Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] Value: "owned"
utils:0153| [stdout] }]
utils:0153| [stdout] }],
utils:0153| [stdout] VolumeType: "gp2"
utils:0153| [stdout] } [recovered]
utils:0153| [stdout] panic:
utils:0153| [stdout]
utils:0153| [stdout] mock: Unexpected Method Call
utils:0153| [stdout] -----------------------------
utils:0153| [stdout]
utils:0153| [stdout] CreateVolume(*ec2.CreateVolumeInput)
utils:0153| [stdout] 0: {
utils:0153| [stdout] AvailabilityZone: "us-east-1a",
utils:0153| [stdout] Encrypted: false,
utils:0153| [stdout] Size: 10,
utils:0153| [stdout] TagSpecifications: [{
utils:0153| [stdout] ResourceType: "volume",
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] Value: "owned"
utils:0153| [stdout] },{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] }]
utils:0153| [stdout] }],
utils:0153| [stdout] VolumeType: "gp2"
utils:0153| [stdout] }
utils:0153| [stdout]
utils:0153| [stdout] The closest call I have is:
utils:0153| [stdout]
utils:0153| [stdout] CreateVolume(*ec2.CreateVolumeInput)
utils:0153| [stdout] 0: {
utils:0153| [stdout] AvailabilityZone: "us-east-1a",
utils:0153| [stdout] Encrypted: false,
utils:0153| [stdout] Size: 10,
utils:0153| [stdout] TagSpecifications: [{
utils:0153| [stdout] ResourceType: "volume",
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] },{
utils:0153| [stdout] Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] Value: "owned"
utils:0153| [stdout] }]
utils:0153| [stdout] }],
utils:0153| [stdout] VolumeType: "gp2"
utils:0153| [stdout] }
utils:0153| [stdout]
utils:0153| [stdout] Difference found in argument 0:
utils:0153| [stdout]
utils:0153| [stdout] --- Expected
utils:0153| [stdout] +++ Actual
utils:0153| [stdout] @@ -7,7 +7,7 @@
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] + Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] + Value: "owned"
utils:0153| [stdout] + },{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] - },{
utils:0153| [stdout] - Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] - Value: "owned"
utils:0153| [stdout] }]
utils:0153| [stdout]
utils:0153| [stdout] Diff: 0: FAIL: {
utils:0153| [stdout] AvailabilityZone: "us-east-1a",
utils:0153| [stdout] Encrypted: false,
utils:0153| [stdout] Size: 10,
utils:0153| [stdout] TagSpecifications: [{
utils:0153| [stdout] ResourceType: "volume",
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] Value: "owned"
utils:0153| [stdout] },{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] }]
utils:0153| [stdout] }],
utils:0153| [stdout] VolumeType: "gp2"
utils:0153| [stdout] } != {
utils:0153| [stdout] AvailabilityZone: "us-east-1a",
utils:0153| [stdout] Encrypted: false,
utils:0153| [stdout] Size: 10,
utils:0153| [stdout] TagSpecifications: [{
utils:0153| [stdout] ResourceType: "volume",
utils:0153| [stdout] Tags: [{
utils:0153| [stdout] Key: "KubernetesCluster",
utils:0153| [stdout] Value: "clusterid.test"
utils:0153| [stdout] },{
utils:0153| [stdout] Key: "kubernetes.io/cluster/clusterid.test",
utils:0153| [stdout] Value: "owned"
utils:0153| [stdout] }]
utils:0153| [stdout] }],
utils:0153| [stdout] VolumeType: "gp2"
utils:0153| [stdout] }
utils:0153| [stdout]
utils:0153| [stdout] goroutine 68 [running]:
utils:0153| [stdout] testing.tRunner.func1.1(0x140c1c0, 0xc0002b6eb0)
utils:0153| [stdout] /usr/lib/go-1.14/src/testing/testing.go:988 +0x30d
utils:0153| [stdout] testing.tRunner.func1(0xc0000d7560)
utils:0153| [stdout] /usr/lib/go-1.14/src/testing/testing.go:991 +0x3f9
utils:0153| [stdout] panic(0x140c1c0, 0xc0002b6eb0)
utils:0153| [stdout] /usr/lib/go-1.14/src/runtime/panic.go:969 +0x166
utils:0153| [stdout] k8s.io/kubernetes/vendor/github.com/stretchr/testify/mock.(*Mock).fail(0xc000459028, 0x17610b3, 0x6f, 0xc000471d00, 0x4, 0x4)
utils:0153| [stdout] /tmp/kubernetes.git/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/stretchr/testify/mock/mock.go:235 +0x17c
utils:0153| [stdout] k8s.io/kubernetes/vendor/github.com/stretchr/testify/mock.(*Mock).MethodCalled(0xc000459028, 0x24cf1ab, 0xc, 0xc0002b6310, 0x1, 0x1, 0xc000471ac0, 0x4, 0x4)
utils:0153| [stdout] /tmp/kubernetes.git/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/stretchr/testify/mock/mock.go:357 +0x90f
utils:0153| [stdout] k8s.io/kubernetes/vendor/github.com/stretchr/testify/mock.(*Mock).Called(0xc000459028, 0xc0002b6310, 0x1, 0x1, 0x0, 0xc000010c08, 0x0)
utils:0153| [stdout] /tmp/kubernetes.git/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/stretchr/testify/mock/mock.go:333 +0x1a7
utils:0153| [stdout] k8s.io/kubernetes/pkg/cloudprovider/providers/aws.(*MockedFakeEC2).CreateVolume(0xc000459020, 0xc000459140, 0x0, 0x0, 0x1)
utils:0153| [stdout] /tmp/kubernetes.git/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws_test.go:72 +0x80
utils:0153| [stdout] k8s.io/kubernetes/pkg/cloudprovider/providers/aws.(*Cloud).CreateDisk(0xc0000d7680, 0xc000019f20, 0x2, 0x2, 0xc0003f55f0, 0x1)
utils:0153| [stdout] /tmp/kubernetes.git/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:2392 +0x535
utils:0153| [stdout] k8s.io/kubernetes/pkg/cloudprovider/providers/aws.TestCreateDisk(0xc0000d7560)
utils:0153| [stdout] /tmp/kubernetes.git/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws_test.go:1837 +0x6aa
utils:0153| [stdout] testing.tRunner(0xc0000d7560, 0x1799aa8)
utils:0153| [stdout] /usr/lib/go-1.14/src/testing/testing.go:1039 +0xdc
utils:0153| [stdout] created by testing.(*T).Run
utils:0153| [stdout] /usr/lib/go-1.14/src/testing/testing.go:1090 +0x372
utils:0153| [stdout] FAIL k8s.io/kubernetes/pkg/cloudprovider/providers/aws 0.140s

See original description

Tags:

Kelsey Steele (kelsey-steele) on 2020-12-18

summary:	- aws test from ubuntu_k8s_unit_tests failed on F-gcp-5.4 + aws test from ubuntu_k8s_unit_tests failed on G-gcp-5.8
description:	updated

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2020-12-29:

I checked again about this failure on all the instances, it looks like this failure is a bit random.

I can see this issue with instance e2-standard-2 and g1-small (passed with n1-standard-8)
But for e2-standard-2 this test passed in the previous run with the same kernel.

g1-small failed with all 2 attempts on the jenkins server.

And since there is no such issue on these 3 instances with the previous kernel (5.8.0-1013.13-gcp), I will retest e2-standard-2 and g1-small again on jenkins.

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2020-12-29:

This test passed with 2 more retests on e2-standard-2 and g1-small.
So this is probably a glitch or something.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.