Search OpenShift CI

#OCPBUGS-15430	issue	13 days ago	KubeAPIDown alert rename and/or degraded status ASSIGNED
We have many guards making sure that there are always at least two instances of the kube-apiserver. If we ever reach a single kube-apiserver and it causes disruption for the clients, other alerts such as KubeAPIErrorBudgetBurn will fire. KubeAPIDown is here to make sure that Prometheus and really any client can reach the kube-apiserver, which they can even when there is only one instance of kube-apiserver running. If they can't or that availability is disrupted, `KubeAPIErrorBudgetBurn` will fire. Comment 23058588 by Marcel Härri at 2023-09-19T06:57:07.949+0000
#OCPBUGS-30267	issue	4 weeks ago	[IBMCloud] MonitorTests liveness/readiness probe error events repeat MODIFIED
Mar 12 18:52:24.937 - 58s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS {alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
pull-ci-openshift-installer-master-e2e-vsphere-upi (all) - 13 runs, 62% failed, 100% of failures match = 62% impact
#1784059398444814336	junit	44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 6m6s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m6s, firing for 0s: Apr 27 04:27:59.959 - 308s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} Apr 27 04:29:09.959 - 58s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1784008378100486144	junit	47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s: Apr 27 01:02:46.928 - 118s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1782534565357686784	junit	6 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 2m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m28s, firing for 0s: Apr 22 23:18:31.881 - 148s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1783969295546454016	junit	2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s: Apr 26 22:28:31.294 - 118s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1783969295546454016	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1783969295546454016	junit	2 days ago
<*errors.errorString \| 0xc001f51a00>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"VSphereOpenshiftClusterHealthFail\",\n \"alertstate\": \"firing\",\n \"check\": \"CheckAccountPermissions\",\n \"container\": \"vsphere-problem-detector-operator\",\n \"endpoint\": \"vsphere-metrics\",\n \"instance\": \"10.130.0.11:8444\",\n \"job\": \"vsphere-problem-detector-metrics\",\n \"namespace\": \"openshift-cluster-storage-operator\",\n \"pod\": \"vsphere-problem-detector-operator-76c59996c4-q59bn\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"vsphere-problem-detector-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714170519.006,\n \"1\"\n ]\n }\n]", },
#1783969295546454016	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1781663856024621056	junit	8 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s: Apr 20 13:40:13.240 - 118s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1780996227094024192	junit	10 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s: Apr 18 17:30:55.412 - 88s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1779931756258922496	junit	13 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 6m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m58s, firing for 0s: Apr 15 19:20:27.206 - 418s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1779931756258922496	junit	13 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1779931756258922496	junit	13 days ago
<*errors.errorString \| 0xc001cc2ac0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"VSphereOpenshiftClusterHealthFail\",\n \"alertstate\": \"firing\",\n \"check\": \"CheckAccountPermissions\",\n \"container\": \"vsphere-problem-detector-operator\",\n \"endpoint\": \"vsphere-metrics\",\n \"instance\": \"10.128.0.15:8444\",\n \"job\": \"vsphere-problem-detector-metrics\",\n \"namespace\": \"openshift-cluster-storage-operator\",\n \"pod\": \"vsphere-problem-detector-operator-5f8bd95dd5-hgpch\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"vsphere-problem-detector-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713208911.842,\n \"1\"\n ]\n }\n]", },
#1779931756258922496	junit	13 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1779853352922779648	junit	13 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1779853352922779648	junit	13 days ago
<*errors.errorString \| 0xc001d589e0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"VSphereOpenshiftClusterHealthFail\",\n \"alertstate\": \"firing\",\n \"check\": \"CheckAccountPermissions\",\n \"container\": \"vsphere-problem-detector-operator\",\n \"endpoint\": \"vsphere-metrics\",\n \"instance\": \"10.129.0.12:8444\",\n \"job\": \"vsphere-problem-detector-metrics\",\n \"namespace\": \"openshift-cluster-storage-operator\",\n \"pod\": \"vsphere-problem-detector-operator-98d97776d-js9kc\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"vsphere-problem-detector-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713188740.393,\n \"1\"\n ]\n }\n]", },
#1779853352922779648	junit	13 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1 [
#1779853352922779648	junit	13 days ago
<*errors.errorString \| 0xc002781230>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|TelemeterClientFailures\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|TargetDown\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"VSphereOpenshiftClusterHealthFail\",\n \"alertstate\": \"firing\",\n \"check\": \"CheckAccountPermissions\",\n \"container\": \"vsphere-problem-detector-operator\",\n \"endpoint\": \"vsphere-metrics\",\n \"instance\": \"10.129.0.12:8444\",\n \"job\": \"vsphere-problem-detector-metrics\",\n \"namespace\": \"openshift-cluster-storage-operator\",\n \"pod\": \"vsphere-problem-detector-operator-98d97776d-js9kc\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"vsphere-problem-detector-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713191410.495,\n \"1\"\n ]\n }\n]", },

Found in 61.54% of runs (100.00% of failures) across 13 total runs and 1 jobs (61.54% failed) in 98ms - clear search | chart view - source code located on github