#OCPBUGS-15430 | issue | 2 weeks ago | KubeAPIDown alert rename and/or degraded status ASSIGNED |
We have many guards making sure that there are always at least two instances of the kube-apiserver. If we ever reach a single kube-apiserver and it causes disruption for the clients, other alerts such as KubeAPIErrorBudgetBurn will fire. KubeAPIDown is here to make sure that Prometheus and really any client can reach the kube-apiserver, which they can even when there is only one instance of kube-apiserver running. If they can't or that availability is disrupted, `KubeAPIErrorBudgetBurn` will fire. Comment 23058588 by Marcel Härri at 2023-09-19T06:57:07.949+0000 | |||
#OCPBUGS-30267 | issue | 2 days ago | [IBMCloud] MonitorTests liveness/readiness probe error events repeat POST |
Mar 12 18:52:24.937 - 58s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS {alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"} | |||
periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-stable-4.12-ocp-e2e-aws-sdn-arm64 (all) - 14 runs, 43% failed, 100% of failures match = 43% impact | |||
#1771787335734661120 | junit | 4 days ago | |
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 4m30s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m30s, firing for 0s: Mar 24 07:08:59.373 - 270s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} | |||
#1771097282540736512 | junit | 6 days ago | |
Mar 22 10:01:59.694 E ns/openshift-cluster-storage-operator pod/cluster-storage-operator-7879f89fbd-hjx22 node/ip-10-0-130-244.us-east-2.compute.internal uid/30d72750-1523-4020-b99e-d7ce891fd725 container/cluster-storage-operator reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar) Mar 22 10:02:04.450 E ns/openshift-insights pod/insights-operator-76fb894b85-rb9n2 node/ip-10-0-130-244.us-east-2.compute.internal uid/ebb20e72-1b55-4234-a841-093e6a6907b9 container/insights-operator reason/ContainerExit code/2 cause/Error g alerts cache for conditional gatherer\nI0322 10:01:58.528786 1 conditional_gatherer.go:278] alert "AlertmanagerReceiversNotConfigured" has state "firing"\nI0322 10:01:58.528802 1 conditional_gatherer.go:278] alert "KubeAPIErrorBudgetBurn" has state "pending"\nI0322 10:01:58.528806 1 conditional_gatherer.go:278] alert "PodSecurityViolation" has state "firing"\nI0322 10:01:58.528811 1 conditional_gatherer.go:278] alert "Watchdog" has state "firing"\nI0322 10:01:58.528869 1 conditional_gatherer.go:288] updating version cache for conditional gatherer\nI0322 10:01:58.532661 1 conditional_gatherer.go:296] cluster version is '4.13.0-0.nightly-arm64-2024-03-22-084957'\nI0322 10:01:58.532694 1 tasks_processing.go:45] number of workers: 1\nI0322 10:01:58.532716 1 tasks_processing.go:69] worker 0 listening for tasks.\nI0322 10:01:58.532720 1 tasks_processing.go:71] worker 0 working on conditional_gatherer_rules task.\nI0322 10:01:58.532764 1 recorder.go:70] Recording insights-operator/conditional-gatherer-rules with fingerprint=8dbbbde181184600277bd0c8401374b23c24c4f4b08634e52ed045ff5aa12179\nI0322 10:01:58.532829 1 gather.go:180] gatherer "conditional" function "conditional_gatherer_rules" took 927ns to process 1 records\nI0322 10:01:58.532776 1 tasks_processing.go:74] worker 0 stopped.\nI0322 10:01:58.532888 1 periodic.go:162] Periodic gather conditional completed in 122ms\nI0322 10:01:58.566806 1 recorder.go:70] Recording insights-operator/gathers with fingerprint=69c20ea5d730385e757505d37eeb7caad8e1d7b698d2b990cfe59ee6ecd4216b\nI0322 10:01:58.566985 1 diskrecorder.go:70] Writing 188 records to /var/lib/insights-operator/insights-2024-03-22-100158.tar.gz\nI0322 10:01:58.580259 1 diskrecorder.go:51] Wrote 188 records to disk in 13ms\nI0322 10:02:01.371123 1 httplog.go:131] "HTTP" verb="GET" URI="/metrics" latency="7.31745ms" userAgent="Prometheus/2.39.1" audit-ID="996c942f-d086-4590-b554-8b076bb3ad81" srcIP="10.128.2.17:49126" resp=200\n Mar 22 10:02:05.688 E ns/openshift-service-ca pod/service-ca-7d78bbf766-589v9 node/ip-10-0-254-217.us-east-2.compute.internal uid/33ca28f8-f5fb-4df1-980e-d227cfc8654c container/service-ca-controller reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar) | |||
#1771097282540736512 | junit | 6 days ago | |
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 53m46s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 53m46s, firing for 0s: Mar 22 09:30:31.518 - 220s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"} Mar 22 09:30:31.518 - 2890s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} Mar 22 09:34:43.518 - 88s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"} Mar 22 10:20:13.518 - 28s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} | |||
#1771335568207122432 | junit | 5 days ago | |
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m36s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m36s, firing for 0s: Mar 23 01:11:57.455 - 96s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} | |||
#1772674209261031424 | junit | 2 days ago | |
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 3m32s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m32s, firing for 0s: Mar 26 17:51:53.126 - 212s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} | |||
#1770347139440840704 | junit | 8 days ago | |
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 1m4s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m4s, firing for 0s: Mar 20 07:48:03.618 - 64s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} | |||
#1768799003022987264 | junit | 12 days ago | |
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending KubeAPIErrorBudgetBurn was at or above pending for at least 3m12s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m12s, firing for 0s: Mar 16 01:18:01.733 - 192s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"} |
Found in 42.86% of runs (100.00% of failures) across 14 total runs and 1 jobs (42.86% failed) in 142ms - clear search | chart view - source code located on github