Search OpenShift CI

#OCPBUGS-30267	issue	5 weeks ago	[IBMCloud] MonitorTests liveness/readiness probe error events repeat MODIFIED
Mar 12 18:52:24.937 - 58s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS {alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
#OCPBUGS-15430	issue	2 weeks ago	KubeAPIDown alert rename and/or degraded status ASSIGNED
We have many guards making sure that there are always at least two instances of the kube-apiserver. If we ever reach a single kube-apiserver and it causes disruption for the clients, other alerts such as KubeAPIErrorBudgetBurn will fire. KubeAPIDown is here to make sure that Prometheus and really any client can reach the kube-apiserver, which they can even when there is only one instance of kube-apiserver running. If they can't or that availability is disrupted, `KubeAPIErrorBudgetBurn` will fire. Comment 23058588 by Marcel Härri at 2023-09-19T06:57:07.949+0000
periodic-ci-openshift-release-master-nightly-4.12-upgrade-from-stable-4.11-e2e-metal-ipi-sdn-bm-upgrade (all) - 20 runs, 75% failed, 27% of failures match = 20% impact
#1785722168873586688	junit	2 days ago
<*errors.errorString \| 0xc0014435a0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"NodeRAIDDegraded\",\n \"alertstate\": \"firing\",\n \"container\": \"kube-rbac-proxy\",\n \"device\": \"md127\",\n \"endpoint\": \"https\",\n \"instance\": \"host5.cluster16.ocpci.eng.rdu2.redhat.com\",\n \"job\": \"node-exporter\",\n \"namespace\": \"openshift-monitoring\",\n \"pod\": \"node-exporter-j6rzt\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"node-exporter\",\n \"severity\": \"critical\"\n },\n \"value\": [\n 1714589762.849,\n \"1\"\n ]\n },\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"NodeRAIDDegraded\",\n \"alertstate\": \"firing\",\n \"container\": \"kube-rbac-proxy\",\n \"device\": \"md127\",\n \"endpoint\": \"https\",\n \"instance\": \"host6...
#1785722168873586688	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1 [
#1783663241499512832	junit	7 days ago
<*errors.errorString \| 0xc001bc2f30>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"NodeRAIDDegraded\",\n \"alertstate\": \"firing\",\n \"container\": \"kube-rbac-proxy\",\n \"device\": \"md127\",\n \"endpoint\": \"https\",\n \"instance\": \"host5.cluster7.ocpci.eng.rdu2.redhat.com\",\n \"job\": \"node-exporter\",\n \"namespace\": \"openshift-monitoring\",\n \"pod\": \"node-exporter-nmngd\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"node-exporter\",\n \"severity\": \"critical\"\n },\n \"value\": [\n 1714098569.087,\n \"1\"\n ]\n },\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"NodeRAIDDegraded\",\n \"alertstate\": \"firing\",\n \"container\": \"kube-rbac-proxy\",\n \"device\": \"md127\",\n \"endpoint\": \"https\",\n \"instance\": \"host6....
#1783663241499512832	junit	7 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1 [
#1784851395652882432	junit	4 days ago
<*errors.errorString \| 0xc00090de50>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"NodeRAIDDegraded\",\n \"alertstate\": \"firing\",\n \"container\": \"kube-rbac-proxy\",\n \"device\": \"md127\",\n \"endpoint\": \"https\",\n \"instance\": \"host5.cluster16.ocpci.eng.rdu2.redhat.com\",\n \"job\": \"node-exporter\",\n \"namespace\": \"openshift-monitoring\",\n \"pod\": \"node-exporter-mjnm7\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"node-exporter\",\n \"severity\": \"critical\"\n },\n \"value\": [\n 1714382378.032,\n \"1\"\n ]\n },\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"NodeRAIDDegraded\",\n \"alertstate\": \"firing\",\n \"container\": \"kube-rbac-proxy\",\n \"device\": \"md127\",\n \"endpoint\": \"https\",\n \"instance\": \"host6...
#1784851395652882432	junit	4 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1 [
#1782839541674020864	junit	9 days ago
<*errors.errorString \| 0xc0012d5660>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"NodeRAIDDegraded\",\n \"alertstate\": \"firing\",\n \"container\": \"kube-rbac-proxy\",\n \"device\": \"md127\",\n \"endpoint\": \"https\",\n \"instance\": \"host5.cluster7.ocpci.eng.rdu2.redhat.com\",\n \"job\": \"node-exporter\",\n \"namespace\": \"openshift-monitoring\",\n \"pod\": \"node-exporter-xrrcg\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"node-exporter\",\n \"severity\": \"critical\"\n },\n \"value\": [\n 1713905949.005,\n \"1\"\n ]\n },\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"NodeRAIDDegraded\",\n \"alertstate\": \"firing\",\n \"container\": \"kube-rbac-proxy\",\n \"device\": \"md127\",\n \"endpoint\": \"https\",\n \"instance\": \"host6....
#1782839541674020864	junit	9 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1 [

Found in 20.00% of runs (26.67% of failures) across 20 total runs and 1 jobs (75.00% failed) in 431ms - clear search | chart view - source code located on github