| #TRT-1664 | issue | 5 days ago | console operator crashing CLOSED |
Issue 15996721: console operator crashing
Description: Undiagnosed panic detected in pod
{code}
{ pods/openshift-console-operator_console-operator-598b8fdbb-v2tnv_console-operator.log.gz:E0510 11:53:50.992299 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
pods/openshift-console-operator_console-operator-598b8fdbb-v2tnv_console-operator.log.gz:E0510 11:53:51.008625 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
pods/openshift-console-operator_console-operator-598b8fdbb-v2tnv_console-operator.log.gz:E0510 11:53:51.023108 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
pods/openshift-console-operator_console-operator-598b8fdbb-v2tnv_console-operator.log.gz:E0510 11:53:51.033921 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
pods/openshift-console-operator_console-operator-598b8fdbb-v2tnv_console-operator.log.gz:E0510 11:53:51.045080 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)}
{code}
[Sippy|https://sippy.dptools.openshift.org/sippy-ng/tests/4.16/analysis?test=Undiagnosed%20panic%20detected%20in%20pod&filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22Undiagnosed%20panic%20detected%20in%20pod%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D] shows this appears to be hitting a few places though it's early and hard to see. Serial looks affected.
[Job runs with failure|https://sippy.dptools.openshift.org/sippy-ng/jobs/4.16/runs?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22failed_test_names%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22Undiagnosed%20panic%20detected%20in%20pod%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D&sortField=timestamp&sort=desc], note due to the nature of where this panic is detected, job runs can pass despite this crash. The failure will show in prow, but the job will can be considered successful if nothing else failed.
[search CI chart view|https://search.dptools.openshift.org/chart?search=console-operator.*Observed+a+panic&maxAge=48h&context=1&type=junit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job] seems to indicate it kicked off roughly 24 hours ago.
Possible revert candidate? https://github.com/openshift/console-operator/pull/895
Status: CLOSED
| |||
| #TRT-1632 | issue | 2 weeks ago | openshift-controller-manager pod panic due to type assertion CLOSED |
Issue 15973057: openshift-controller-manager pod panic due to type assertion
Description: Caught by the test: Undiagnosed panic detected in pod
Sample job run:
[https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-upgrade/1783981854974545920]
Error message
{code}
{ pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:02.367266 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)
pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:03.368403 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)
pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:04.370157 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)}
{code}
[Sippy indicates|https://sippy.dptools.openshift.org/sippy-ng/tests/4.16/analysis?test=Undiagnosed%20panic%20detected%20in%20pod&filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22Undiagnosed%20panic%20detected%20in%20pod%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D] it's happening a small percentage of the time since around Apr 25th.
Took out the last payload so labeling trt-incident for now.
See the linked OCPBUG for the actual component.
Status: CLOSED
| |||
| #OCPBUGS-18138 | issue | 2 weeks ago | CI sees: pods/openshift-console-operator_console-operator-6bf69485c8-2s498_console-operator.log.gz:E0821 06:29:09.418195 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" Verified |
Issue 15443283: CI sees: pods/openshift-console-operator_console-operator-6bf69485c8-2s498_console-operator.log.gz:E0821 06:29:09.418195 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference"
Description: Description of problem:
{code:none}
example job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-jboss-fuse-camel-k-test-container-main-camel-k-ocp4.14-lp-interop-camel-k-interop-aws/1693503447845834752
: Undiagnosed panic detected in pod expand_less0s{ pods/openshift-console-operator_console-operator-6bf69485c8-2s498_console-operator.log.gz:E0821 06:29:09.418195 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)}{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
intermittent. But seems to happen every few days or so: $ podman run -it corbinu/alpine-w3m -dump -cols 200 "https://search.ci.openshift.org/?search=openshift-console-operator_console-operator.*invalid+memory+address+or+nil+pointer+dereference&maxAge=336h&context=1&type=junit&name=.*4.14.*&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job"
[openshift-console-op] [14d] [Links ] [junit ]
[Search]
Job:
[.*4.14.* ] [ ] [5 ] [20971520 ] [job ]
[ ] Wrap lines
periodic-ci-jboss-fuse-camel-k-test-container-main-camel-k-ocp4.14-lp-interop-camel-k-interop-aws (all) - 2 runs, 0% failed, 50% of runs match
#1693503447845834752 junit 4 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-2s498_console-operator.log.gz:E0821 06:29:09.418195 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-cpu-partitioning (all) - 54 runs, 7% failed, 25% of failures match = 2% impact
#1692398384507260928 junit 7 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-tppxt_console-operator.log.gz:E0818 05:14:35.489038 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-fips-serial (all) - 2 runs, 0% failed, 50% of runs match
#1692131892385550336 junit 8 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-lr54b_console-operator.log.gz:E0817 11:36:17.381961 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-nightly-4.14-e2e-gcp-ovn-upi (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1692132131523792896 junit 8 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-ztc7h_console-operator.log.gz:E0817 11:48:41.058274 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-shared-vpc-phz-techpreview (all) - 42 runs, 12% failed, 20% of failures match = 2% impact
#1691881846972878848 junit 8 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-ppmnm_console-operator.log.gz:E0816 19:12:20.716903 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-knative-serverless-operator-main-ocp4.14-lp-interop-operator-e2e-interop-aws-ocp414 (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
#1691835547812630528 junit 9 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-rm6fk_console-operator.log.gz:E0816 16:04:44.717384 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-nightly-4.14-e2e-gcp-ovn-rt (all) - 24 runs, 21% failed, 20% of failures match = 4% impact
#1691779459859877888 junit 9 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6df6498794-wqjk6_console-operator.log.gz:E0816 12:19:02.860700 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-cluster-etcd-operator-release-4.14-periodics-e2e-aws-etcd-recovery (all) - 14 runs, 100% failed, 7% of failures match = 7% impact
#1691303819960389632 junit 10 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-9f9fc5c5f-54xpb_console-operator_previous.log.gz:E0815 04:49:27.747727 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-proxy (all) - 34 runs, 24% failed, 13% of failures match = 3% impact
#1691023830476132352 junit 11 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6849dd7f6b-4r246_console-operator.log.gz:E0814 10:21:13.247983 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn (all) - 29 runs, 28% failed, 13% of failures match = 3% impact
#1690908928604377088 junit 11 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-bfcc5bfc7-qndlf_console-operator.log.gz:E0814 03:03:21.007675 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-csi (all) - 24 runs, 13% failed, 33% of failures match = 4% impact
#1690909013211877376 junit 11 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-bfcc5bfc7-6g886_console-operator.log.gz:E0814 02:43:39.815176 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-zones (all) - 2 runs, 0% failed, 50% of runs match
#1690604928210309120 junit 12 days ago
# Undiagnosed panic detected in pod
pods/openshift-console-operator_console-operator-6bf69485c8-sfnjf_console-operator.log.gz:E0813 06:29:07.539267 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Found in 0.10% of runs (0.24% of failures) across 12026 total runs and 737 jobs (41.81% failed) in 1.057s - clear search | chart view - source code located on github
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-jboss-fuse-camel-k-test-container-main-camel-k-ocp4.14-lp-interop-camel-k-interop-aws/1693503447845834752/artifacts/camel-k-interop-aws/gather-extra/artifacts/pods/openshift-console-operator_console-operator-6bf69485c8-2s498_console-operator.log has the stack trace:
E0821 06:29:09.418195 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 262 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x26aba20?, 0x475f720})
/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x733dad?})
/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x26aba20, 0x475f720})
/usr/lib/golang/src/runtime/panic.go:884 +0x213
github.com/openshift/console-operator/pkg/console/controllers/util.IncludeNamesFilter.func1({0x27daba0?, 0xc00045a900?})
/go/src/github.com/openshift/console-operator/pkg/console/controllers/util/util.go:34 +0x16f
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete({0xc0005e82d0?, {0x2f66350?, 0xc000d21b18?}}, {0x27daba0, 0xc00045a900})
/go/src/github.com/openshift/console-operator/vendor/k8s.io/client-go/tools/cache/controller.go:327 +0x46
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/openshift/console-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:978 +0xaf
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x475d67?, {0x2f3e8e0, 0xc000da8b40}, 0x1, 0xc000df8060)
/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc000fe8788?)
/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000309710)
/go/src/github.com/openshift/console-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:967 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/openshift/console-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x85{code}
Status: Verified
| |||
| #TRT-1643 | issue | 9 days ago | Another controller manager undiagnosed panic capable of failing payloads CLOSED |
Issue 15982864: Another controller manager undiagnosed panic capable of failing payloads
Description: Similar to https://issues.redhat.com/browse/TRT-1632, this job failed a 4.17 nightly payload last night when one of ten jobs hit this crash:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.17-e2e-aws-ovn-upgrade/1786172902819762176
{code}
: Undiagnosed panic detected in pod expand_less 0s
{ pods/openshift-controller-manager_controller-manager-6f79ddc977-fcnm7_controller-manager.log.gz:E0503 01:49:29.881473 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c65060), concrete:(*abi.Type)(0x3e5bc20), asserted:(*abi.Type)(0x418ee20), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.BuildConfig)}
{code}
Status: CLOSED
| |||
| #OCPBUGS-33088 | issue | 7 days ago | openshift-controller-manager pod panic due to type assertion ON_QA |
Issue 15973055: openshift-controller-manager pod panic due to type assertion
Description: Caught by the test: Undiagnosed panic detected in pod
Sample job run:
[https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-upgrade/1783981854974545920]
Error message
{code}
{ pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:02.367266 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)
pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:03.368403 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)
pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:04.370157 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)}
{code}
[Sippy indicates|https://sippy.dptools.openshift.org/sippy-ng/tests/4.16/analysis?test=Undiagnosed%20panic%20detected%20in%20pod&filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22Undiagnosed%20panic%20detected%20in%20pod%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D] it's happening a small percentage of the time since around Apr 25th.
Took out the last payload so labeling trt-incident for now.
Status: Verified
| |||
| #OCPBUGS-33371 | issue | 37 hours ago | openshift-controller-manager pod panic due to type assertion POST |
Issue 15989657: openshift-controller-manager pod panic due to type assertion
Description: This is a clone of issue OCPBUGS-33088. The following is the description of the original issue: --- Caught by the test: Undiagnosed panic detected in pod
Sample job run:
[https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-upgrade/1783981854974545920]
Error message
{code}
{ pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:02.367266 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)
pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:03.368403 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)
pods/openshift-controller-manager_controller-manager-6b66bf5587-6ghjk_controller-manager.log.gz:E0426 23:06:04.370157 1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x3c6a2a0), concrete:(*abi.Type)(0x3e612c0), asserted:(*abi.Type)(0x419cdc0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Secret)}
{code}
[Sippy indicates|https://sippy.dptools.openshift.org/sippy-ng/tests/4.16/analysis?test=Undiagnosed%20panic%20detected%20in%20pod&filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22Undiagnosed%20panic%20detected%20in%20pod%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D] it's happening a small percentage of the time since around Apr 25th.
Took out the last payload so labeling trt-incident for now.
Status: POST
| |||
| periodic-ci-openshift-release-master-stable-4.8-upgrade-from-stable-4.4-e2e-aws-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact | |||
| #1790198797809225728 | junit | 2 days ago | |
# Undiagnosed panic detected in pod pods/openshift-authentication-operator_authentication-operator-8685f84874-hgk6x_operator_previous.log.gz:E0514 02:24:55.429365 1 runtime.go:78] Observed a panic: "AuthenticationOperator2: timed out waiting for caches to sync" (AuthenticationOperator2: timed out waiting for caches to sync) | |||
Found in 100.00% of runs (100.00% of failures) across 1 total runs and 1 jobs (100.00% failed) in 86ms - clear search | chart view - source code located on github