Search OpenShift CI

#OCPBUGS-15430	issue	2 weeks ago	KubeAPIDown alert rename and/or degraded status ASSIGNED
We have many guards making sure that there are always at least two instances of the kube-apiserver. If we ever reach a single kube-apiserver and it causes disruption for the clients, other alerts such as KubeAPIErrorBudgetBurn will fire. KubeAPIDown is here to make sure that Prometheus and really any client can reach the kube-apiserver, which they can even when there is only one instance of kube-apiserver running. If they can't or that availability is disrupted, `KubeAPIErrorBudgetBurn` will fire. Comment 23058588 by Marcel Härri at 2023-09-19T06:57:07.949+0000
#OCPBUGS-30267	issue	4 weeks ago	[IBMCloud] MonitorTests liveness/readiness probe error events repeat MODIFIED
Mar 12 18:52:24.937 - 58s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS {alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-cpu-partitioning (all) - 56 runs, 100% failed, 27% of failures match = 27% impact
#1784735653586538496	junit	5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 3m22s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m22s, firing for 0s: Apr 29 00:41:16.008 - 202s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1784554178878640128	junit	17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 4m20s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m20s, firing for 0s: Apr 28 12:44:26.934 - 260s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1783919248108687360	junit	2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 3m16s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m16s, firing for 0s: Apr 26 18:36:06.545 - 196s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1783737396710346752	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable",alertstate="firing",severity!="info"} >= 1 [
#1783737396710346752	junit	2 days ago
<*errors.errorString \| 0xc001eaaff0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"OperatorHubSourceError\",\n \"alertstate\": \"firing\",\n \"container\": \"catalog-operator\",\n \"endpoint\": \"https-metrics\",\n \"exported_namespace\": \"openshift-marketplace\",\n \"instance\": \"10.128.0.18:8443\",\n \"job\": \"catalog-operator-metrics\",\n \"name\": \"certified-operators\",\n \"namespace\": \"openshift-operator-lifecycle-manager\",\n \"pod\": \"catalog-operator-6f9f8bdbb6-9rrhq\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"catalog-operator-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714113611.751,\n \"1\"\n ]\n }\n]", },
#1783737396710346752	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable",alertstate="firing",severity!="info"} >= 1 [
#1783737396710346752	junit	2 days ago
<*errors.errorString \| 0xc000b39530>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"OperatorHubSourceError\",\n \"alertstate\": \"firing\",\n \"container\": \"catalog-operator\",\n \"endpoint\": \"https-metrics\",\n \"exported_namespace\": \"openshift-marketplace\",\n \"instance\": \"10.128.0.18:8443\",\n \"job\": \"catalog-operator-metrics\",\n \"name\": \"certified-operators\",\n \"namespace\": \"openshift-operator-lifecycle-manager\",\n \"pod\": \"catalog-operator-6f9f8bdbb6-9rrhq\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"catalog-operator-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714116473.252,\n \"1\"\n ]\n }\n]", },
#1783828318554951680	junit	2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m14s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m14s, firing for 0s: Apr 26 12:37:08.297 - 74s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1783828318554951680	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable",alertstate="firing",severity!="info"} >= 1 [
#1783828318554951680	junit	2 days ago
<*errors.errorString \| 0xc000d66c30>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"OperatorHubSourceError\",\n \"alertstate\": \"firing\",\n \"container\": \"catalog-operator\",\n \"endpoint\": \"https-metrics\",\n \"exported_namespace\": \"openshift-marketplace\",\n \"instance\": \"10.130.0.19:8443\",\n \"job\": \"catalog-operator-metrics\",\n \"name\": \"certified-operators\",\n \"namespace\": \"openshift-operator-lifecycle-manager\",\n \"pod\": \"catalog-operator-5c4c5b595b-bbv9r\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"catalog-operator-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714135175.341,\n \"1\"\n ]\n }\n]", },
#1783828318554951680	junit	2 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable",alertstate="firing",severity!="info"} >= 1 [
#1783646725404102656	junit	3 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m16s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m16s, firing for 0s: Apr 26 00:37:42.706 - 76s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1782920502742880256	junit	5 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 5m26s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m26s, firing for 0s: Apr 24 00:29:36.865 - 326s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1782920502742880256	junit	5 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable",alertstate="firing",severity!="info"} >= 1 [
#1782920502742880256	junit	5 days ago
<*errors.errorString \| 0xc001a56bd0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|KubeMemoryOvercommit\|KubeMemoryOvercommit\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|OVNKubernetesResourceRetryFailure\|OVNKubernetesResourceRetryFailure\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\|TechPreviewNoUpgrade\|ClusterNotUpgradeable\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"OperatorHubSourceError\",\n \"alertstate\": \"firing\",\n \"container\": \"catalog-operator\",\n \"endpoint\": \"https-metrics\",\n \"exported_namespace\": \"openshift-marketplace\",\n \"instance\": \"10.129.0.29:8443\",\n \"job\": \"catalog-operator-metrics\",\n \"name\": \"certified-operators\",\n \"namespace\": \"openshift-operator-lifecycle-manager\",\n \"pod\": \"catalog-operator-7bd787cd9c-9vbpp\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"catalog-operator-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1713918770.393,\n \"1\"\n ]\n }\n]", },
#1782285572648210432	junit	6 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 5m56s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m56s, firing for 0s: Apr 22 06:25:57.732 - 356s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1782194722597507072	junit	7 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 5m36s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m36s, firing for 0s: Apr 22 00:23:25.182 - 278s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} Apr 22 00:28:35.182 - 58s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1782104124679000064	junit	7 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m6s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m6s, firing for 0s: Apr 21 18:23:31.525 - 66s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1781197121337167872	junit	9 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 46s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 46s, firing for 0s: Apr 19 06:21:02.606 - 46s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1780743376627306496	junit	11 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 3m58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m58s, firing for 0s: Apr 18 00:17:39.136 - 238s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1780199171488223232	junit	12 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 34s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 34s, firing for 0s: Apr 16 12:14:23.487 - 34s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1780289769410924544	junit	12 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 3m8s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m8s, firing for 0s: Apr 16 18:14:51.316 - 188s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1780108321487851520	junit	13 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 24s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 24s, firing for 0s: Apr 16 06:14:48.524 - 24s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}

Found in 26.79% of runs (26.79% of failures) across 56 total runs and 1 jobs (100.00% failed) in 143ms - clear search | chart view - source code located on github