Search OpenShift CI

#OCPBUGS-15430	issue	13 days ago	KubeAPIDown alert rename and/or degraded status ASSIGNED
We have many guards making sure that there are always at least two instances of the kube-apiserver. If we ever reach a single kube-apiserver and it causes disruption for the clients, other alerts such as KubeAPIErrorBudgetBurn will fire. KubeAPIDown is here to make sure that Prometheus and really any client can reach the kube-apiserver, which they can even when there is only one instance of kube-apiserver running. If they can't or that availability is disrupted, `KubeAPIErrorBudgetBurn` will fire. Comment 23058588 by Marcel Härri at 2023-09-19T06:57:07.949+0000
#OCPBUGS-30267	issue	4 weeks ago	[IBMCloud] MonitorTests liveness/readiness probe error events repeat MODIFIED
Mar 12 18:52:24.937 - 58s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS {alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-telco5g (all) - 31 runs, 16% failed, 200% of failures match = 32% impact
#1783677600233689088	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1783677600233689088	junit	2 days ago
<*errors.errorString \| 0xc00146f5d0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714103400.695,\n \"1\"\n ]\n }\n]", },
#1783677600233689088	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1783677600233689088	junit	2 days ago
<*errors.errorString \| 0xc001abd0c0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714106096.882,\n \"1\"\n ]\n }\n]", },
#1783315138527367168	junit	3 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1783315138527367168	junit	3 days ago
<*errors.errorString \| 0xc0015409a0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714016853.953,\n \"1\"\n ]\n }\n]", },
#1783315138527367168	junit	3 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1783315138527367168	junit	3 days ago
<*errors.errorString \| 0xc001f5c090>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714019799.524,\n \"1\"\n ]\n }\n]", },
#1782346945885376512	junit	6 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1782346945885376512	junit	6 days ago
<*errors.errorString \| 0xc0011f2a00>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713786908.542,\n \"1\"\n ]\n }\n]", },
#1782346945885376512	junit	6 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1782346945885376512	junit	6 days ago
<*errors.errorString \| 0xc002432cd0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713789650.911,\n \"1\"\n ]\n }\n]", },
#1780778451293179904	junit	10 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780778451293179904	junit	10 days ago
<*errors.errorString \| 0xc001ecca20>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713411726.195,\n \"1\"\n ]\n }\n]", },
#1780778451293179904	junit	10 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780778451293179904	junit	10 days ago
<*errors.errorString \| 0xc001201010>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713414620.346,\n \"1\"\n ]\n }\n]", },
#1780416218927206400	junit	11 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780416218927206400	junit	11 days ago
<*errors.errorString \| 0xc001ab82f0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713325719.529,\n \"1\"\n ]\n }\n]", },
#1780416218927206400	junit	11 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780416218927206400	junit	11 days ago
<*errors.errorString \| 0xc0022c5e10>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713328541.07,\n \"1\"\n ]\n }\n]", },
#1780963375774175232	junit	10 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780963375774175232	junit	10 days ago
<*errors.errorString \| 0xc001a453c0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713456222.874,\n \"1\"\n ]\n }\n]", },
#1780963375774175232	junit	10 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780963375774175232	junit	10 days ago
<*errors.errorString \| 0xc00242fe90>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713459157.415,\n \"1\"\n ]\n }\n]", },
#1780152271598784512	junit	12 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780152271598784512	junit	12 days ago
<*errors.errorString \| 0xc000ea8de0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713262229.023,\n \"1\"\n ]\n }\n]", },
#1780152271598784512	junit	12 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780152271598784512	junit	12 days ago
<*errors.errorString \| 0xc0010f1570>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713265153.518,\n \"1\"\n ]\n }\n]", },
#1780325185648332800	junit	12 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780325185648332800	junit	12 days ago
<*errors.errorString \| 0xc0016b8b80>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713303291.608,\n \"1\"\n ]\n }\n]", },
#1780325185648332800	junit	12 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780325185648332800	junit	12 days ago
<*errors.errorString \| 0xc00108ee00>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713306168.197,\n \"1\"\n ]\n }\n]", },
#1780052690433216512	junit	12 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780052690433216512	junit	12 days ago
<*errors.errorString \| 0xc001bb1150>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713238754.489,\n \"1\"\n ]\n }\n]", },
#1780052690433216512	junit	12 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1780052690433216512	junit	12 days ago
<*errors.errorString \| 0xc0015f11c0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713241421.909,\n \"1\"\n ]\n }\n]", },
#1779691227394871296	junit	13 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1779691227394871296	junit	13 days ago
<*errors.errorString \| 0xc001da0cb0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713152652.05,\n \"1\"\n ]\n }\n]", },
#1779691227394871296	junit	13 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1779691227394871296	junit	13 days ago
<*errors.errorString \| 0xc0020fef90>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"CannotRetrieveUpdates\",\n \"alertstate\": \"firing\",\n \"namespace\": \"openshift-cluster-version\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713155423.026,\n \"1\"\n ]\n }\n]", },

Found in 32.26% of runs (200.00% of failures) across 31 total runs and 1 jobs (16.13% failed) in 125ms - clear search | chart view - source code located on github