Search OpenShift CI

#OCPBUGS-15430	issue	3 weeks ago	KubeAPIDown alert rename and/or degraded status ASSIGNED
We have many guards making sure that there are always at least two instances of the kube-apiserver. If we ever reach a single kube-apiserver and it causes disruption for the clients, other alerts such as KubeAPIErrorBudgetBurn will fire. KubeAPIDown is here to make sure that Prometheus and really any client can reach the kube-apiserver, which they can even when there is only one instance of kube-apiserver running. If they can't or that availability is disrupted, `KubeAPIErrorBudgetBurn` will fire. Comment 23058588 by Marcel Härri at 2023-09-19T06:57:07.949+0000
#OCPBUGS-30267	issue	5 weeks ago	[IBMCloud] MonitorTests liveness/readiness probe error events repeat MODIFIED
Mar 12 18:52:24.937 - 58s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS {alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-sdn-upgrade (all) - 19 runs, 26% failed, 80% of failures match = 21% impact
#1786077053892694016	junit	4 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 4m12s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m12s, firing for 0s: May 02 17:27:10.150 - 252s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1785344531449778176	junit	6 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 3m48s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m48s, firing for 0s: Apr 30 17:02:17.265 - 228s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1783713668211412992	junit	11 days ago
<*errors.errorString \| 0xc001c9e960>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"OperatorHubSourceError\",\n \"alertstate\": \"firing\",\n \"container\": \"catalog-operator\",\n \"endpoint\": \"https-metrics\",\n \"exported_namespace\": \"openshift-marketplace\",\n \"instance\": \"10.129.0.36:8443\",\n \"job\": \"catalog-operator-metrics\",\n \"name\": \"redhat-marketplace\",\n \"namespace\": \"openshift-operator-lifecycle-manager\",\n \"pod\": \"catalog-operator-7845c8d48d-lntsh\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"catalog-operator-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714111349.011,\n \"1\"\n ]\n }\n]", },
#1783713668211412992	junit	11 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1 [
#1783713668211412992	junit	11 days ago
<*errors.errorString \| 0xc001c0d4f0>{ s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n {\n \"metric\": {\n \"__name__\": \"ALERTS\",\n \"alertname\": \"OperatorHubSourceError\",\n \"alertstate\": \"firing\",\n \"container\": \"catalog-operator\",\n \"endpoint\": \"https-metrics\",\n \"exported_namespace\": \"openshift-marketplace\",\n \"instance\": \"10.129.0.36:8443\",\n \"job\": \"catalog-operator-metrics\",\n \"name\": \"redhat-marketplace\",\n \"namespace\": \"openshift-operator-lifecycle-manager\",\n \"pod\": \"catalog-operator-7845c8d48d-lntsh\",\n \"prometheus\": \"openshift-monitoring/k8s\",\n \"service\": \"catalog-operator-metrics\",\n \"severity\": \"warning\"\n },\n \"value\": [\n 1714113683.419,\n \"1\"\n ]\n }\n]", },
#1783713668211412992	junit	11 days ago
promQL query returned unexpected results: ALERTS{alertname!~"Watchdog\|AlertmanagerReceiversNotConfigured\|PrometheusRemoteWriteDesiredShards\|KubeJobFailed\|Watchdog\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|KubePodNotReady\|etcdMembersDown\|etcdMembersDown\|etcdGRPCRequestsSlow\|etcdGRPCRequestsSlow\|etcdHighNumberOfFailedGRPCRequests\|etcdHighNumberOfFailedGRPCRequests\|etcdMemberCommunicationSlow\|etcdMemberCommunicationSlow\|etcdNoLeader\|etcdNoLeader\|etcdHighFsyncDurations\|etcdHighFsyncDurations\|etcdHighCommitDurations\|etcdHighCommitDurations\|etcdInsufficientMembers\|etcdInsufficientMembers\|etcdHighNumberOfLeaderChanges\|etcdHighNumberOfLeaderChanges\|KubeAPIErrorBudgetBurn\|KubeAPIErrorBudgetBurn\|KubeClientErrors\|KubeClientErrors\|KubePersistentVolumeErrors\|KubePersistentVolumeErrors\|MCDDrainError\|MCDDrainError\|MCDPivotError\|MCDPivotError\|PrometheusOperatorWatchErrors\|PrometheusOperatorWatchErrors\|RedhatOperatorsCatalogError\|RedhatOperatorsCatalogError\|VSphereOpenshiftNodeHealthFail\|VSphereOpenshiftNodeHealthFail\|SamplesImagestreamImportFailing\|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1 [
#1783548733913305088	junit	11 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 38s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 38s, firing for 0s: Apr 25 18:02:35.300 - 38s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}

Found in 21.05% of runs (80.00% of failures) across 19 total runs and 1 jobs (26.32% failed) in 190ms - clear search | chart view - source code located on github