Job:
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-sdn-serial (all) - 1 runs, 0% failed, 100% of runs match
#1773309993714454528junit14 minutes ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 12:25:29.748 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-network-migration-rollback (all) - 1 runs, 0% failed, 100% of runs match
#1773309998726647808junit18 minutes ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m10s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m10s, firing for 0s:
Mar 28 13:03:15.344 - 190s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-heavy-build-ovn-remote-libvirt-s390x (all) - 2 runs, 0% failed, 100% of runs match
#1773334428714536960junit42 minutes ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 13:41:38.724 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772971993260363776junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 13:47:49.222 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-vsphere-ovn-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773310237529346048junit46 minutes ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m28s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m28s, firing for 0s:
Mar 28 12:45:37.169 - 448s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-cilium (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773309991315312640junit45 minutes ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 28 12:18:44.935 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-csi-operator-master-e2e-azure-ovn-upgrade (all) - 7 runs, 43% failed, 67% of failures match = 29% impact
#1773312157874655232junit47 minutes ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 12:33:22.356 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772961436042006528junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m56s, firing for 0s:
Mar 27 13:29:05.986 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:53:35.986 - 328s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-sdn (all) - 1 runs, 0% failed, 100% of runs match
#1773309992149979136junit53 minutes ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 28 12:28:30.065 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-ovirt-sdn-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773310237499985920junit59 minutes ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h28m42s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"ovirt", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1h28m42s, firing for 0s:
Mar 28 12:19:23.618 - 1350s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 12:19:23.618 - 3914s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 12:46:03.618 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
pull-ci-openshift-api-master-e2e-upgrade-minor (all) - 26 runs, 19% failed, 20% of failures match = 4% impact
#1773304446474784768junitAbout an hour ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 37m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 37m58s, firing for 0s:
Mar 28 11:51:32.645 - 2278s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-aws-sdn-imdsv2 (all) - 1 runs, 0% failed, 100% of runs match
#1773309987125202944junitAbout an hour ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m8s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 5m8s, firing for 0s:
Mar 28 12:03:01.209 - 308s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
release-openshift-origin-installer-e2e-azure-shared-vpc-4.16 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773296648370262016junit2 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m12s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m12s, firing for 0s:
Mar 28 11:36:16.619 - 284s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 11:36:32.619 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-upgrade-from-nightly-4.15-ocp-ovn-remote-libvirt-s390x (all) - 4 runs, 25% failed, 300% of failures match = 75% impact
#1773304219718127616junit2 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m24s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m24s, firing for 0s:
Mar 28 12:32:16.735 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 12:38:16.735 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 12:43:16.735 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773123028410961920junit13 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m18s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m18s, firing for 0s:
Mar 28 00:48:52.011 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 00:53:52.011 - 350s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772941784771465216junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m26s, firing for 0s:
Mar 27 12:54:53.194 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 12:59:53.194 - 418s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-ibmcloud-csi (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773310250099675136junit2 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 13m26s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 13m26s, firing for 0s:
Mar 28 12:24:34.493 - 628s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 12:25:04.493 - 178s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-ovn-upgrade (all) - 8 runs, 38% failed, 67% of failures match = 25% impact
#1773280555815669760junit2 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 28 10:34:06.199 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773150708883787776junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 21m48s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 21m48s, firing for 0s:
Mar 28 02:02:46.559 - 182s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 02:02:46.559 - 1126s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773150708883787776junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m44s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m44s, firing for 0s:
periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn-upgrade (all) - 7 runs, 14% failed, 400% of failures match = 57% impact
#1773280556625170432junit2 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m12s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m12s, firing for 0s:
Mar 28 11:27:28.969 - 312s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773280556625170432junit2 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h26m24s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h26m24s, firing for 0s:
Mar 28 10:18:04.312 - 1176s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 10:18:04.312 - 3456s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 11:17:12.312 - 552s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773249179200524288junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m28s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m28s, firing for 0s:
Mar 28 07:57:45.427 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773216254727819264junit7 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m4s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m4s, firing for 0s:
Mar 28 05:46:01.581 - 124s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773058325101940736junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m10s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m10s, firing for 0s:
Mar 27 19:28:29.162 - 14s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:29:45.162 - 148s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:33:45.162 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-azure-sdn-upgrade (all) - 10 runs, 60% failed, 167% of failures match = 100% impact
#1773278840500523008junit2 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 29m10s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 29m10s, firing for 0s:
Mar 28 10:09:06.983 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 10:38:12.983 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 10:42:42.983 - 868s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 10:57:42.983 - 358s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 11:04:12.983 - 408s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773223828726484992junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 56s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 56s, firing for 0s:
Mar 28 06:32:15.929 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 07:19:31.929 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773166083155431424junit9 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m8s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 12m8s, firing for 0s:
Mar 28 04:00:52.619 - 728s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773166083155431424junit9 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h5m56s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1h5m56s, firing for 0s:
Mar 28 02:51:52.838 - 1520s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 03:18:44.838 - 2436s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773110867919900672junit13 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 23:11:00.837 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773057850516443136junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m26s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 7m26s, firing for 0s:
Mar 27 19:32:34.621 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:05:26.621 - 238s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773008873112735744junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m56s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 8m56s, firing for 0s:
Mar 27 16:20:34.859 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 16:54:28.859 - 328s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772958855928483840junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m18s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28m18s, firing for 0s:
Mar 27 12:56:20.054 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:23:30.054 - 268s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:30:30.054 - 598s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:42:30.054 - 298s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:48:00.054 - 298s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:56:30.054 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772810506663366656junit33 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 03:11:53.026 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772719853233246208junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m56s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 12m56s, firing for 0s:
Mar 26 21:05:20.417 - 238s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 21:36:46.417 - 538s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772629178051465216junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 32m4s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 32m4s, firing for 0s:
Mar 26 15:09:24.950 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 15:35:54.950 - 330s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 15:43:26.950 - 868s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 15:58:26.950 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 16:00:26.950 - 670s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-sdn-serial-aws-arm64 (all) - 8 runs, 13% failed, 300% of failures match = 38% impact
#1773280461548687360junit2 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 25m46s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 25m46s, firing for 0s:
Mar 28 10:04:56.475 - 128s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 10:04:56.475 - 1418s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773189391938752512junit9 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m22s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 7m22s, firing for 0s:
Mar 28 04:02:25.616 - 442s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772733085566111746junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m54s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m54s, firing for 0s:
Mar 26 22:19:33.085 - 174s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
release-openshift-origin-installer-e2e-azure-upgrade (all) - 9 runs, 11% failed, 400% of failures match = 44% impact
#1773280540888141824junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 18m2s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 18m2s, firing for 0s:
Mar 28 11:25:27.487 - 148s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 11:25:27.487 - 758s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 11:25:41.487 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 28 11:29:27.487 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1773280541005582336junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 28 11:19:56.685 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773280547431256064junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 28 11:20:38.137 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773280541798305792junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 11m4s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 11m4s, firing for 0s:
Mar 28 11:12:05.648 - 606s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 11:16:05.648 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-azure-sdn-upgrade (all) - 70 runs, 21% failed, 460% of failures match = 99% impact
#1773251923017732096junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 24m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 24m56s, firing for 0s:
Mar 28 09:14:11.285 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:21:11.285 - 1408s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251917149900800junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m8s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m8s, firing for 0s:
Mar 28 09:54:04.725 - 248s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251917149900800junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30m38s, firing for 0s:
Mar 28 09:03:43.528 - 1838s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251920551481344junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m38s, firing for 0s:
Mar 28 09:52:23.258 - 278s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251920551481344junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 41m6s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 41m6s, firing for 0s:
Mar 28 08:35:33.256 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 08:58:03.256 - 2108s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251916311040000junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m18s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 5m18s, firing for 0s:
Mar 28 09:43:50.991 - 318s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251916311040000junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 36m24s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 36m24s, firing for 0s:
Mar 28 08:41:41.800 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:06:41.800 - 1348s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:29:41.800 - 598s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251921344204800junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30s, firing for 0s:
Mar 28 09:51:02.059 - 30s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251921344204800junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 26m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 26m56s, firing for 0s:
Mar 28 09:13:04.295 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:19:34.295 - 1408s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251918810845184junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m10s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m10s, firing for 0s:
Mar 28 09:52:40.617 - 190s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251918810845184junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 32m30s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 32m30s, firing for 0s:
Mar 28 08:38:52.966 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:01:52.966 - 1138s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:21:52.966 - 724s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251919737786368junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m42s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m42s, firing for 0s:
Mar 28 09:39:03.405 - 282s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251919737786368junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 32m46s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 32m46s, firing for 0s:
Mar 28 08:38:48.052 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:03:48.052 - 1318s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:26:18.052 - 500s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251915472179200junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 6m16s, firing for 0s:
Mar 28 09:39:13.298 - 258s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:45:03.298 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251915472179200junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 54m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 54m58s, firing for 0s:
#1773251917988761600junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m38s, firing for 0s:
Mar 28 09:33:52.036 - 218s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251917988761600junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m22s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28m22s, firing for 0s:
Mar 28 09:01:02.894 - 1702s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251922187259904junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m8s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m8s, firing for 0s:
Mar 28 09:34:18.200 - 128s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773251922187259904junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30m28s, firing for 0s:
Mar 28 08:59:58.050 - 1828s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138633931362304junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 38m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 38m16s, firing for 0s:
Mar 28 01:07:03.396 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:12:33.396 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:30:33.396 - 1820s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138631423168512junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 34m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 34m16s, firing for 0s:
Mar 28 01:06:47.845 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:30:17.845 - 928s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:46:17.845 - 1010s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138630580113408junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m44s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m44s, firing for 0s:
Mar 28 02:11:49.173 - 164s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138630580113408junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m20s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28m20s, firing for 0s:
Mar 28 01:34:33.240 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:35:33.240 - 1672s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138632262029312junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 10m16s, firing for 0s:
Mar 28 02:16:52.121 - 616s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138632262029312junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 55m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 55m54s, firing for 0s:
Mar 28 01:17:41.105 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 01:17:41.105 - 3296s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138635613278208junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m10s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m10s, firing for 0s:
Mar 28 02:07:17.822 - 70s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138635613278208junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 35m18s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 35m18s, firing for 0s:
Mar 28 01:05:29.861 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:29:29.861 - 1048s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:47:29.861 - 922s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138634774417408junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m32s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m32s, firing for 0s:
Mar 28 02:14:01.530 - 152s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138634774417408junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 34m4s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 34m4s, firing for 0s:
Mar 28 01:35:36.050 - 2044s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138638134054912junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m4s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m4s, firing for 0s:
Mar 28 02:14:39.590 - 184s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138638134054912junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 45m42s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 45m42s, firing for 0s:
Mar 28 01:00:44.950 - 628s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:11:44.950 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:24:14.950 - 1876s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138637299388416junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m24s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m24s, firing for 0s:
Mar 28 02:11:42.500 - 144s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138637299388416junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 25m50s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 25m50s, firing for 0s:
Mar 28 01:41:38.672 - 1550s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138633092501504junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m30s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m30s, firing for 0s:
Mar 28 02:05:07.898 - 270s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138633092501504junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m12s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30m12s, firing for 0s:
Mar 28 01:06:39.155 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:09:09.155 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:10:39.155 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:30:09.155 - 928s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:46:39.155 - 770s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138636456333312junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m30s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m30s, firing for 0s:
Mar 28 02:05:29.267 - 150s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773138636456333312junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 29m34s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 29m34s, firing for 0s:
Mar 28 01:27:30.415 - 1774s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047458583023616junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m16s, firing for 0s:
Mar 27 20:28:07.789 - 76s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047458583023616junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 29m24s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 29m24s, firing for 0s:
Mar 27 19:28:56.616 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:53:26.616 - 1048s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:11:56.616 - 628s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047457765134336junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m10s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m10s, firing for 0s:
Mar 27 20:21:17.201 - 190s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047457765134336junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 36m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 36m28s, firing for 0s:
Mar 27 19:04:28.983 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:28:58.983 - 1980s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047460281716736junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m34s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m34s, firing for 0s:
Mar 27 20:24:30.768 - 154s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047460281716736junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 29m0s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 29m0s, firing for 0s:
Mar 27 19:49:36.470 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:51:06.470 - 1712s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047456074829824junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m20s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m20s, firing for 0s:
Mar 27 20:18:21.806 - 200s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047456074829824junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 43m34s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 43m34s, firing for 0s:
Mar 27 19:04:12.710 - 838s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:28:42.710 - 1018s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:46:42.710 - 758s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047459421884416junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m48s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m48s, firing for 0s:
Mar 27 20:05:57.998 - 228s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047459421884416junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 34m14s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 34m14s, firing for 0s:
Mar 27 19:06:49.548 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:10:19.548 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:30:19.548 - 1348s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:53:49.548 - 470s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047461095411712junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 29m22s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 29m22s, firing for 0s:
Mar 27 19:32:17.355 - 328s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:39:17.355 - 1434s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047457324732416junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 35m8s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 35m8s, firing for 0s:
Mar 27 19:11:08.945 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:17:38.945 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:35:38.945 - 1048s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:53:38.945 - 674s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047454464217088junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m16s, firing for 0s:
Mar 27 20:13:10.448 - 256s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047454464217088junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 56m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 56m58s, firing for 0s:
Mar 27 19:04:27.722 - 868s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:20:27.722 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:21:57.722 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:27:27.722 - 2464s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047452706803712junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m32s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m32s, firing for 0s:
Mar 27 20:11:43.542 - 34s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:13:49.542 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047452706803712junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 35m14s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 35m14s, firing for 0s:
#1773047455261134848junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m14s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m14s, firing for 0s:
Mar 27 20:03:31.117 - 194s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773047455261134848junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 36m4s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 36m4s, firing for 0s:
Mar 27 19:02:17.609 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:06:47.609 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:26:17.609 - 1348s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:49:47.609 - 520s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714695688192junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m0s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m0s, firing for 0s:
Mar 27 12:38:34.780 - 180s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714695688192junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 35m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 35m38s, firing for 0s:
Mar 27 11:39:06.819 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:45:06.819 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 12:04:06.819 - 1288s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 12:26:06.819 - 494s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714775379968junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m40s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m40s, firing for 0s:
Mar 27 12:54:08.824 - 280s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714775379968junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28m38s, firing for 0s:
Mar 27 12:20:50.762 - 1718s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714943152128junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 27m40s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 27m40s, firing for 0s:
Mar 27 12:01:31.714 - 1660s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714590830592junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 10m54s, firing for 0s:
Mar 27 12:35:33.082 - 654s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714590830592junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 31m24s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 31m24s, firing for 0s:
Mar 27 11:53:27.427 - 1884s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714540498944junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m22s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 5m22s, firing for 0s:
Mar 27 12:36:48.681 - 322s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714540498944junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 44m12s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 44m12s, firing for 0s:
Mar 27 11:32:41.113 - 838s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:57:11.113 - 1814s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714813128704junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m32s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 6m32s, firing for 0s:
Mar 27 12:32:57.166 - 392s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714813128704junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 44m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 44m56s, firing for 0s:
Mar 27 11:35:30.822 - 598s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:47:00.822 - 298s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:52:30.822 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:58:30.822 - 1772s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714905403392junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m8s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m8s, firing for 0s:
Mar 27 12:35:24.711 - 188s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714905403392junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 41m46s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 41m46s, firing for 0s:
Mar 27 11:31:04.428 - 568s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:55:04.428 - 1938s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714741825536junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m40s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m40s, firing for 0s:
Mar 27 12:33:09.441 - 280s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714741825536junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m44s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30m44s, firing for 0s:
Mar 27 11:32:51.644 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:57:51.644 - 1078s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 12:16:51.644 - 708s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714850877440junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m50s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 6m50s, firing for 0s:
Mar 27 12:30:47.223 - 410s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714850877440junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 34m6s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 34m6s, firing for 0s:
Mar 27 11:52:07.844 - 2046s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714636967936junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 29m36s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 29m36s, firing for 0s:
Mar 27 11:27:51.920 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:51:51.920 - 1498s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 12:18:21.920 - 190s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772933714636967936junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m52s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m52s, firing for 0s:
#1772842871469117440junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m34s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m34s, firing for 0s:
Mar 27 06:45:33.951 - 154s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842871469117440junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m8s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30m8s, firing for 0s:
Mar 27 05:56:39.436 - 1808s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842875575341056junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m22s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m22s, firing for 0s:
Mar 27 06:32:40.850 - 82s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842875575341056junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 38m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 38m16s, firing for 0s:
Mar 27 05:28:34.312 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:34:34.312 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:54:04.312 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:55:34.312 - 958s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:12:04.312 - 954s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842870605090816junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m16s, firing for 0s:
Mar 27 06:27:22.462 - 136s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842870605090816junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30m56s, firing for 0s:
Mar 27 05:27:39.911 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:51:39.911 - 1558s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:19:39.911 - 180s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842869770424320junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 8m26s, firing for 0s:
Mar 27 06:08:58.985 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:15:28.985 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842868931563520junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m28s, firing for 0s:
Mar 27 06:29:52.343 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842868931563520junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 25m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 25m26s, firing for 0s:
Mar 27 05:26:52.057 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:34:22.057 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:53:22.057 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:57:52.057 - 598s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:08:22.057 - 784s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842873012621312junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m6s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m6s, firing for 0s:
Mar 27 06:30:43.563 - 186s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842873012621312junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 40m18s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 40m18s, firing for 0s:
Mar 27 05:29:49.936 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:40:19.936 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:53:49.936 - 1138s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:13:19.936 - 774s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842873851482112junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m30s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m30s, firing for 0s:
Mar 27 06:25:12.866 - 270s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842873851482112junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m20s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28m20s, firing for 0s:
Mar 27 05:52:15.987 - 1700s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842868096897024junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m14s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m14s, firing for 0s:
Mar 27 06:25:01.423 - 134s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842868096897024junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 40m2s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 40m2s, firing for 0s:
Mar 27 05:27:46.557 - 628s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:50:46.557 - 1774s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842874686148608junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m20s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m20s, firing for 0s:
Mar 27 06:28:51.990 - 140s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842874686148608junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m44s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28m44s, firing for 0s:
Mar 27 05:55:44.312 - 1724s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842877211119616junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m44s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m44s, firing for 0s:
Mar 27 06:28:38.985 - 164s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772842877211119616junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 26m2s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 26m2s, firing for 0s:
Mar 27 05:28:55.130 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:53:55.130 - 1408s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:18:55.130 - 96s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752598705639424junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m36s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m36s, firing for 0s:
Mar 27 00:55:47.568 - 156s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752598705639424junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 43m52s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 43m52s, firing for 0s:
Mar 26 23:47:25.243 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:12:55.243 - 2274s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752599854878720junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 23m2s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 23m2s, firing for 0s:
Mar 27 00:06:24.917 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:11:24.917 - 1324s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752602887360512junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 38m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 38m16s, firing for 0s:
Mar 26 23:40:35.363 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:05:05.363 - 1078s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:23:35.363 - 770s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752602044305408junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m26s, firing for 0s:
Mar 27 00:35:10.609 - 266s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752602044305408junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 47m40s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 47m40s, firing for 0s:
Mar 26 23:33:08.778 - 748s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:55:38.778 - 2112s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752597032112128junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m46s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m46s, firing for 0s:
Mar 27 00:34:58.076 - 286s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752597032112128junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 33m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 33m28s, firing for 0s:
Mar 26 23:35:15.534 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:58:15.534 - 1708s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:27:15.534 - 182s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752596168085504junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m28s, firing for 0s:
Mar 27 00:31:13.251 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752596168085504junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 34m14s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 34m14s, firing for 0s:
Mar 26 23:52:12.342 - 2054s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752601238999040junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m8s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m8s, firing for 0s:
Mar 27 00:27:26.450 - 188s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752601238999040junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 40m6s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 40m6s, firing for 0s:
Mar 26 23:27:37.177 - 538s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:51:07.177 - 1078s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:09:37.177 - 790s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752600484024320junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m58s, firing for 0s:
Mar 27 00:36:21.887 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752600484024320junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30m38s, firing for 0s:
Mar 27 00:01:22.444 - 1838s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752603722027008junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m46s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m46s, firing for 0s:
Mar 27 00:23:32.607 - 286s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752603722027008junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 35m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 35m16s, firing for 0s:
Mar 26 23:26:50.879 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:34:20.879 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:37:20.879 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:51:20.879 - 928s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:07:20.879 - 684s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752597866778624junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 26m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 26m38s, firing for 0s:
Mar 26 23:48:55.013 - 1048s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:07:25.013 - 538s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:17:55.013 - 12s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660990232498176junit42 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 14m2s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 14m2s, firing for 0s:
Mar 26 18:34:32.436 - 842s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660990232498176junit42 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 31m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 31m26s, firing for 0s:
Mar 26 17:56:05.392 - 1886s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660992744886272junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m54s, firing for 0s:
Mar 26 18:40:16.552 - 234s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660992744886272junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 29m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 29m54s, firing for 0s:
Mar 26 18:05:41.687 - 1794s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660995244691456junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m28s, firing for 0s:
Mar 26 18:31:37.353 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660995244691456junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 48m42s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 48m42s, firing for 0s:
Mar 26 17:28:36.625 - 838s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:52:36.625 - 2084s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660996091940864junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m58s, firing for 0s:
Mar 26 18:23:38.917 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660996091940864junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 35m10s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 35m10s, firing for 0s:
Mar 26 17:23:38.772 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:27:38.772 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:47:08.772 - 1078s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 18:05:38.772 - 796s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660989389443072junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m48s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28m48s, firing for 0s:
Mar 26 17:45:00.020 - 1728s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660996964356096junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m50s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m50s, firing for 0s:
Mar 26 18:25:56.827 - 110s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660996964356096junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 31m44s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 31m44s, firing for 0s:
Mar 26 17:26:47.712 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:51:47.712 - 1018s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 18:09:17.712 - 708s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660993583747072junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m40s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 5m40s, firing for 0s:
Mar 26 18:31:43.403 - 340s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660993583747072junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m20s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30m20s, firing for 0s:
Mar 26 17:52:55.640 - 1820s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660994393247744junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m16s, firing for 0s:
Mar 26 18:31:32.362 - 256s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660994393247744junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 31m16s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 31m16s, firing for 0s:
Mar 26 17:34:49.511 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:38:49.511 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:58:19.511 - 1760s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660997752885248junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m38s, firing for 0s:
Mar 26 18:22:06.391 - 218s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660997752885248junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 32m18s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 32m18s, firing for 0s:
Mar 26 17:45:16.205 - 1938s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-nightly-4.13-ocp-ovn-remote-libvirt-s390x (all) - 4 runs, 100% failed, 100% of failures match = 100% impact
#1773274021274587136junit3 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m26s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m26s, firing for 0s:
Mar 28 10:44:16.298 - 298s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 10:50:16.298 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773092781816811520junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m56s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m56s, firing for 0s:
Mar 27 23:06:24.386 - 148s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 23:10:54.386 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772911587141619712junit27 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m26s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m26s, firing for 0s:
Mar 27 10:54:12.921 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 10:59:12.921 - 148s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772730392462233600junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m48s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m48s, firing for 0s:
Mar 26 22:58:25.907 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:00:25.907 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:02:25.907 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:03:55.907 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:07:55.907 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:09:55.907 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-ibmcloud-ovn-multi-s390x (all) - 4 runs, 100% failed, 100% of failures match = 100% impact
#1773263176687685632junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m56s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28m56s, firing for 0s:
Mar 28 09:33:33.335 - 1678s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:35:33.335 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1773081937770975232junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 51m12s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 51m12s, firing for 0s:
Mar 27 21:35:40.736 - 50s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 21:37:02.736 - 2308s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 21:37:32.736 - 538s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 22:17:02.736 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:23:02.736 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772900740667281408junit28 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 31m56s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 31m56s, firing for 0s:
Mar 27 09:34:44.343 - 1408s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 09:35:14.343 - 508s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1772719544754769920junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m28s, firing for 0s:
Mar 26 21:40:04.626 - 568s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-nightly-4.13-ocp-ovn-remote-libvirt-ppc64le (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
#1773274020339257344junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"libvirt", Architecture:"ppc64le", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 28 10:39:21.081 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-gcp-ovn-upgrade (all) - 70 runs, 27% failed, 11% of failures match = 3% impact
#1773251902335619072junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 45m8s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 45m8s, firing for 0s:
Mar 28 08:16:53.209 - 290s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 08:16:53.209 - 2390s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 08:23:15.209 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1773047481240653824junit17 hours ago
        <*errors.errorString | 0xc000f0de30>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.build.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711565050.983,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.image.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711565050.983,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.oauth.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711565050.983,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.packages.operators.coreos.com\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711565050.983,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.route.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711565050.983,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.user.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711565050.983,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1beta1.metrics.k8s.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711565050.983,\n      \"1\"\n    ]\n  }\n]",
        },
#1773047481240653824junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 59m12s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 59m12s, firing for 0s:
Mar 27 18:41:49.568 - 444s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 18:41:49.568 - 2094s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 18:50:45.568 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 19:18:15.568 - 568s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:35:45.568 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-upgrade (all) - 5 runs, 0% failed, 40% of runs match
#1773249168249196544junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m56s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m56s, firing for 0s:
Mar 28 08:06:25.661 - 176s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772730496946540544junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 22s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 22s, firing for 0s:
Mar 26 21:40:51.934 - 22s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-ovn-remote-libvirt-s390x-heterogeneous (all) - 4 runs, 50% failed, 100% of failures match = 50% impact
#1773258927446691840junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 8m58s, firing for 0s:
Mar 28 09:21:03.786 - 538s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773077684285345792junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h4m32s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h4m32s, firing for 0s:
Mar 27 21:12:01.729 - 3872s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-stable-4.13-ocp-e2e-aws-sdn-arm64 (all) - 6 runs, 50% failed, 67% of failures match = 33% impact
#1773245681281011712junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m40s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m40s, firing for 0s:
Mar 28 07:45:19.135 - 160s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773189392853110784junit7 hours ago
promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1773189392853110784junit7 hours ago
        <*errors.errorString | 0xc0018cb150>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"ClusterOperatorDegraded\",\n      \"alertstate\": \"firing\",\n      \"name\": \"monitoring\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711604849.02,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"ClusterOperatorDown\",\n      \"alertstate\": \"firing\",\n      \"name\": \"monitoring\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"UpdatingPrometheusK8SFailed\",\n      \"severity\": \"critical\"\n    },\n    \"value\": [\n      1711604849.02,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeDeploymentReplicasMismatch\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-main\",\n      \"deployment\": \"image-registry\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-image-registry\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711604849.02,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodNotScheduled\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-main\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-image-registry\",\n      \"pod\": \"image-registry-755f998988-rfbmt\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"a3d99757-485d-48bd-a4b4-3c83cbea2d61\"\n    },\n    \"value\": [\n      1711604849.02,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeStatefulSetReplicasMismatch\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-main\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-monitoring\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"statefulset\": \"prometheus-k8s\"\n    },\n    \"value\": [\n      1711604849.02,\n      \"1\"\n    ]\n  }\n]",
        },
#1773189392853110784junit7 hours ago
        <*errors.errorString | 0xc00157d880>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"ClusterMonitoringOperatorReconciliationErrors\",\n      \"alertstate\": \"firing\",\n      \"container\": \"cluster-monitoring-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.47:8443\",\n      \"job\": \"cluster-monitoring-operator\",\n      \"namespace\": \"openshift-monitoring\",\n      \"pod\": \"cluster-monitoring-operator-584dddbbfd-cqj9c\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"cluster-monitoring-operator\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"ClusterOperatorDegraded\",\n      \"alertstate\": \"firing\",\n      \"name\": \"monitoring\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"ClusterOperatorDown\",\n      \"alertstate\": \"firing\",\n      \"name\": \"monitoring\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"UpdatingPrometheusK8SFailed\",\n      \"severity\": \"critical\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeContainerWaiting\",\n      \"alertstate\": \"firing\",\n      \"container\": \"config-reloader\",\n      \"namespace\": \"openshift-monitoring\",\n      \"pod\": \"prometheus-k8s-0\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeContainerWaiting\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy\",\n      \"namespace\": \"openshift-monitoring\",\n      \"pod\": \"prometheus-k8s-0\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeContainerWaiting\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-thanos\",\n      \"namespace\": \"openshift-monitoring\",\n      \"pod\": \"prometheus-k8s-0\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeContainerWaiting\",\n      \"alertstate\": \"firing\",\n      \"container\": \"prometheus\",\n      \"namespace\": \"openshift-monitoring\",\n      \"pod\": \"prometheus-k8s-0\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeContainerWaiting\",\n      \"alertstate\": \"firing\",\n      \"container\": \"prometheus-proxy\",\n      \"namespace\": \"openshift-monitoring\",\n      \"pod\": \"prometheus-k8s-0\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeContainerWaiting\",\n      \"alertstate\": \"firing\",\n      \"container\": \"thanos-sidecar\",\n      \"namespace\": \"openshift-monitoring\",\n      \"pod\": \"prometheus-k8s-0\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeDeploymentReplicasMismatch\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-main\",\n      \"deployment\": \"image-registry\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-image-registry\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711608153.836,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodNotScheduled\",\n  ...
periodic-ci-openshift-release-master-ci-4.13-e2e-gcp-secureboot (all) - 1 runs, 0% failed, 100% of runs match
#1773264939016785920junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m24s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m24s, firing for 0s:
Mar 28 09:02:47.989 - 118s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 09:02:47.989 - 178s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 09:07:17.989 - 148s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-altinfra-e2e-vsphere-capi-ovn (all) - 11 runs, 100% failed, 100% of failures match = 100% impact
#1773258837386596352junit5 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773258837386596352junit5 hours ago
        <*errors.errorString | 0xc00071ed10>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.31:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7849f6879d-5nn8g\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711616538.727,\n      \"1\"\n    ]\n  }\n]",
        },
#1773258837386596352junit5 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773258837386596352junit5 hours ago
        <*errors.errorString | 0xc00321e110>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.31:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7849f6879d-5nn8g\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711619197.716,\n      \"1\"\n    ]\n  }\n]",
        },
#1773198346404499456junit9 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773198346404499456junit9 hours ago
        <*errors.errorString | 0xc0013655a0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.28:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6f849b8f98-flh2d\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711601857.94,\n      \"1\"\n    ]\n  }\n]",
        },
#1773198346404499456junit9 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773198346404499456junit9 hours ago
        <*errors.errorString | 0xc00250d0e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.28:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6f849b8f98-flh2d\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711604416.142,\n      \"1\"\n    ]\n  }\n]",
        },
#1773137957222354944junit13 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m58s, firing for 0s:
Mar 28 01:02:33.325 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773137957222354944junit13 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773137957222354944junit13 hours ago
        <*errors.errorString | 0xc001a0b430>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.25:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-5755f69b54-5hm7d\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711587767.233,\n      \"1\"\n    ]\n  }\n]",
        },
#1773137957222354944junit13 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773077571882192896junit17 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773077571882192896junit17 hours ago
        <*errors.errorString | 0xc00235f650>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.14:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-794cb5b8c9-p72vl\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711573153.652,\n      \"1\"\n    ]\n  }\n]",
        },
#1773077571882192896junit17 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773077571882192896junit17 hours ago
        <*errors.errorString | 0xc0023cf130>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.14:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-794cb5b8c9-p72vl\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711575916.047,\n      \"1\"\n    ]\n  }\n]",
        },
#1773017154736623616junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 17:01:24.362 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773017154736623616junit20 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773017154736623616junit20 hours ago
        <*errors.errorString | 0xc002394f50>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.34:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-67898b8df-pxkfx\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711558882.118,\n      \"1\"\n    ]\n  }\n]",
        },
#1773017154736623616junit20 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772956761938989056junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 13:01:04.615 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772956761938989056junit25 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772956761938989056junit25 hours ago
        <*errors.errorString | 0xc003281fb0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.14:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-fd67f9795-lcrwh\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711544447.873,\n      \"1\"\n    ]\n  }\n]",
        },
#1772956761938989056junit25 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772896495163412480junit29 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772896495163412480junit29 hours ago
        <*errors.errorString | 0xc0022f5f60>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.13:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-866bf6bc4c-4nnfk\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711530003.617,\n      \"1\"\n    ]\n  }\n]",
        },
#1772896495163412480junit29 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772896495163412480junit29 hours ago
        <*errors.errorString | 0xc00081cb50>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.13:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-866bf6bc4c-4nnfk\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711532944.871,\n      \"1\"\n    ]\n  }\n]",
        },
#1772835958572453888junit33 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772835958572453888junit33 hours ago
        <*errors.errorString | 0xc001ca3440>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.9:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-8c4ff984f-fv29c\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711515696.961,\n      \"1\"\n    ]\n  }\n]",
        },
#1772835958572453888junit33 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772835958572453888junit33 hours ago
        <*errors.errorString | 0xc00118fdd0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.9:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-8c4ff984f-fv29c\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711518148.676,\n      \"1\"\n    ]\n  }\n]",
        },
#1772775572896747520junit37 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772775572896747520junit37 hours ago
        <*errors.errorString | 0xc003399fd0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.26:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-645b5c745f-rz67l\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711501265.02,\n      \"1\"\n    ]\n  }\n]",
        },
#1772775572896747520junit37 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772775572896747520junit37 hours ago
        <*errors.errorString | 0xc0021d1e60>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.26:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-645b5c745f-rz67l\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711503904.502,\n      \"1\"\n    ]\n  }\n]",
        },
#1772715225586339840junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 26 20:58:51.179 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772715225586339840junit41 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772715225586339840junit41 hours ago
        <*errors.errorString | 0xc001d34f50>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.23:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-757dc9c9f-q9dcj\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711489568.246,\n      \"1\"\n    ]\n  }\n]",
        },
#1772654786525532160junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 26 17:04:20.918 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772654786525532160junit45 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772654786525532160junit45 hours ago
        <*errors.errorString | 0xc00133b1a0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.33:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-5cdc858d47-klr2t\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711472679.815,\n      \"1\"\n    ]\n  }\n]",
        },
#1772654786525532160junit45 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-stable-4.14-ocp-e2e-upgrade-gcp-ovn-heterogeneous (all) - 8 runs, 25% failed, 50% of failures match = 13% impact
#1773253587573411840junit4 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 14m10s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 14m10s, firing for 0s:
Mar 28 08:29:42.917 - 522s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 08:39:56.917 - 328s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-azure-sdn (all) - 5 runs, 0% failed, 20% of runs match
#1773249142324203520junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 28 08:23:44.982 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-gcp-sdn-techpreview (all) - 5 runs, 20% failed, 200% of failures match = 40% impact
#1773249156513533952junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 11m32s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 11m32s, firing for 0s:
Mar 28 08:08:55.298 - 196s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 08:08:55.298 - 496s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619012409135104junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m36s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 9m36s, firing for 0s:
Mar 26 14:17:59.932 - 168s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 14:17:59.932 - 408s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-ovn-upi-serial (all) - 5 runs, 40% failed, 200% of failures match = 80% impact
#1773249148938620928junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m56s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 8m56s, firing for 0s:
Mar 28 07:55:19.768 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 07:55:19.768 - 508s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773150743079948288junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m56s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m56s, firing for 0s:
Mar 28 01:27:57.970 - 148s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:31:57.970 - 148s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772730490143379456junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m28s, firing for 0s:
Mar 26 21:35:42.666 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619045065986048junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m28s, firing for 0s:
Mar 26 14:16:47.343 - 568s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-ovn-techpreview (all) - 5 runs, 20% failed, 400% of failures match = 80% impact
#1773249139149115392junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 28 08:14:29.402 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773058352390082560junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m56s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10m56s, firing for 0s:
Mar 27 19:34:54.264 - 598s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:36:24.264 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1772730521210589184junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 26 21:56:46.566 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619038359293952junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 26 14:36:50.823 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-sdn-cgroupsv2 (all) - 6 runs, 17% failed, 200% of failures match = 33% impact
#1773249146422038528junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m12s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m12s, firing for 0s:
Mar 28 08:03:48.056 - 72s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772757619811815424junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m28s, firing for 0s:
Mar 26 23:23:46.520 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-sdn (all) - 6 runs, 17% failed, 500% of failures match = 83% impact
#1773249145591566336junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m16s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m16s, firing for 0s:
Mar 28 07:59:19.734 - 256s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773150743931392000junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m54s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m54s, firing for 0s:
Mar 28 01:27:59.058 - 174s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773086495947624448junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m36s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m36s, firing for 0s:
Mar 27 21:13:48.566 - 216s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773058363307855872junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m46s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 7m18s, firing for 2m28s:
Mar 27 19:26:13.726 - 50s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 19:26:13.726 - 50s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:29:31.726 - 34s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 19:29:31.726 - 304s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:27:03.726 - 148s  E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772730522007506944junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m0s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m0s, firing for 0s:
Mar 26 22:15:45.918 - 180s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-sdn (all) - 6 runs, 17% failed, 400% of failures match = 67% impact
#1773249153980174336junit5 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 08:09:56.393 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773150748138278912junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 01:33:11.859 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773058367569268736junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 19:30:54.138 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619060995952640junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 26 14:33:02.055 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-ovn-upi (all) - 5 runs, 20% failed, 300% of failures match = 60% impact
#1773249198448185344junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 07:57:03.956 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773150721269567488junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 28 01:24:15.840 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773058341480697856junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 19:18:44.818 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-aws-sdn-arm64 (all) - 6 runs, 0% failed, 33% of runs match
#1773245682845487104junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m42s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m42s, firing for 0s:
Mar 28 07:48:18.058 - 162s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773189389602525184junit10 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m10s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m10s, firing for 0s:
Mar 28 04:00:01.683 - 190s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.16-e2e-openstack-externallb (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773246819069857792junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"openstack", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m28s, firing for 0s:
Mar 28 08:01:14.429 - 388s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-ovn-csi (all) - 5 runs, 0% failed, 40% of runs match
#1773249158182866944junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 28 08:15:40.197 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773058371675492352junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 19:37:18.525 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-csi (all) - 5 runs, 0% failed, 80% of runs match
#1773249154877755392junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 17m44s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 17m44s, firing for 0s:
Mar 28 08:02:48.020 - 312s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 08:02:48.020 - 752s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773150753985138688junit13 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 18s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 18s, firing for 0s:
Mar 28 01:29:19.239 - 18s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772730524528283648junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m28s, firing for 0s:
Mar 26 22:18:31.494 - 268s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619044235513856junit2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 14m48s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 14m48s, firing for 0s:
Mar 26 14:20:52.122 - 48s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 14:20:52.122 - 840s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-agnostic-ovn-cmd (all) - 12 runs, 83% failed, 50% of failures match = 42% impact
#1773241259784146944junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m58s, firing for 0s:
Mar 28 07:53:36.984 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773106751478960128junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 23:17:04.083 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773062516553814016junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m28s, firing for 0s:
Mar 27 20:18:10.639 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773053732305506304junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m28s, firing for 0s:
Mar 27 19:32:34.952 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772993600154505216junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 15:36:21.124 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-azure-sdn-upgrade (all) - 11 runs, 45% failed, 160% of failures match = 73% impact
#1773211592100417536junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 16m28s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 16m28s, firing for 0s:
Mar 28 05:54:33.534 - 988s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773075606360035328junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m56s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m56s, firing for 0s:
Mar 27 20:47:07.251 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:51:37.251 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772986061463490560junit21 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 14:56:29.411 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772878875315933184junit28 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 07:54:53.664 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772825548565778432junit32 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m56s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 8m56s, firing for 0s:
Mar 27 04:25:30.531 - 508s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 04:35:30.531 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772746033223176192junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m28s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 7m28s, firing for 0s:
Mar 26 23:01:51.173 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772691724791975936junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m10s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 8m10s, firing for 0s:
Mar 26 19:24:56.623 - 490s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772639543413968896junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 56s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 56s, firing for 0s:
Mar 26 16:02:29.058 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 16:03:29.058 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-aws-csi (all) - 11 runs, 27% failed, 67% of failures match = 18% impact
#1773241259826089984junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m28s, firing for 0s:
Mar 28 07:40:52.118 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773093091390001152junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4s, firing for 0s:
Mar 27 21:46:40.455 - 4s    I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-ovn (all) - 23 runs, 0% failed, 48% of runs match
#1773225428475973632junit6 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 06:47:24.693 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773189440399740928junit9 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 28 04:24:18.349 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773150708980256768junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 28 02:04:51.644 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773130048224628736junit13 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 28 00:28:37.390 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773095778277920768junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 22:09:59.348 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773058325533954048junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 19:55:04.786 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773052240806809600junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 19:23:34.087 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773016002825555968junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 17:01:43.449 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772981020983824384junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 14:34:35.043 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772870290095738880junit30 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 07:19:05.990 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772653358281134080junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 26 16:58:26.304 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-ovn-fips-serial (all) - 2 runs, 0% failed, 50% of runs match
#1773218884715810816junit7 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 34s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 34s, firing for 0s:
Mar 28 06:02:25.614 - 34s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ibmcloud-ovn-multi-s390x (all) - 4 runs, 75% failed, 133% of failures match = 100% impact
#1773217877575340032junit7 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 13m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 13m28s, firing for 0s:
Mar 28 06:28:13.886 - 808s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773036637408727040junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 22m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 22m28s, firing for 0s:
Mar 27 18:32:30.957 - 1348s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772855441684959232junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 21m56s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 21m56s, firing for 0s:
Mar 27 06:32:54.883 - 1048s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:33:24.883 - 268s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1772674245764059136junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 14m26s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 14m26s, firing for 0s:
Mar 26 18:27:36.781 - 658s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 18:28:06.781 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-ovn-remote-libvirt-s390x (all) - 9 runs, 100% failed, 89% of failures match = 89% impact
#1773213634458030080junit7 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 18m32s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 18m32s, firing for 0s:
Mar 28 05:56:46.922 - 1112s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773097029975674880junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 8m58s, firing for 0s:
Mar 27 22:07:41.662 - 538s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773065262338150400junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 25m24s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 25m24s, firing for 0s:
Mar 27 19:49:04.307 - 778s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:07:04.307 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:49:34.307 - 688s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773031641967497216junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 45m54s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 45m54s, firing for 0s:
Mar 27 17:35:58.775 - 2548s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 18:18:58.775 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 18:44:28.775 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773003781819076608junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m28s, firing for 0s:
Mar 27 15:45:14.674 - 388s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967069717893120junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m26s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 12m26s, firing for 0s:
Mar 27 13:30:49.501 - 388s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:39:49.501 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772866008239837184junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m28s, firing for 0s:
Mar 27 06:41:24.191 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772851195820380160junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m24s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 12m24s, firing for 0s:
Mar 27 05:50:16.853 - 686s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:52:14.853 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
pull-ci-openshift-machine-config-operator-release-4.12-e2e-aws (all) - 1 runs, 0% failed, 100% of runs match
#1773225127840845824junit7 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m40s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 1m40s, firing for 0s:
Mar 28 06:28:07.919 - 100s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-kubernetes-master-e2e-agnostic-ovn-cmd (all) - 12 runs, 92% failed, 18% of failures match = 17% impact
#1773216950613184512junit8 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m28s, firing for 0s:
Mar 28 06:34:02.148 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772986659361525760junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 15:22:07.411 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-oc-master-e2e-agnostic-ovn-cmd (all) - 8 runs, 88% failed, 29% of failures match = 25% impact
#1773214133227884544junit8 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 28 06:26:16.969 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773203483302825984junit8 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 05:38:05.213 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-api-master-e2e-aws-ovn-techpreview (all) - 26 runs, 23% failed, 17% of failures match = 4% impact
#1773199514597855232junit9 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 26s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 26s, firing for 0s:
Mar 28 04:45:24.123 - 26s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-cluster-etcd-operator-release-4.14-periodics-e2e-aws-etcd-recovery (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
#1773203532804001792junit9 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h5m58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 47m2s, firing for 18m56s:
Mar 28 05:04:35.588 - 360s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 05:05:35.588 - 300s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 05:06:05.588 - 270s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 28 05:13:33.588 - 452s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 05:13:33.588 - 452s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 05:37:03.588 - 494s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 05:37:03.588 - 494s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 05:10:35.588 - 178s  E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
Mar 28 05:21:05.588 - 958s  E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772841097001701376junit33 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h1m2s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 36m36s, firing for 24m26s:
Mar 27 05:01:42.247 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:02:12.247 - 178s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 05:02:40.247 - 150s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 27 05:03:10.247 - 120s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
Mar 27 05:12:08.247 - 332s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 05:12:08.247 - 332s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:35:08.247 - 438s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 05:35:08.247 - 438s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:05:10.247 - 418s  E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
Mar 27 05:17:40.247 - 1048s E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-aws-ovn-arm64-techpreview-serial (all) - 6 runs, 33% failed, 100% of failures match = 33% impact
#1773189389564776448junit9 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 14m20s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 14m20s, firing for 0s:
Mar 28 04:00:29.143 - 860s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772632551358205952junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m54s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m54s, firing for 0s:
Mar 26 15:12:39.638 - 234s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-aws-ovn-arm64-techpreview (all) - 6 runs, 33% failed, 100% of failures match = 33% impact
#1773189389644468224junit9 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 38s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 38s, firing for 0s:
Mar 28 04:01:29.001 - 38s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772733733481222144junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m12s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m12s, firing for 0s:
Mar 26 21:50:55.133 - 192s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-openstack-ovn (all) - 15 runs, 20% failed, 33% of failures match = 7% impact
#1773164762511708160junit10 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m42s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"openstack", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m42s, firing for 0s:
Mar 28 02:57:45.770 - 462s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade (all) - 2 runs, 0% failed, 100% of runs match
#1773156167636226048junit10 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m14s on platformidentification.JobType{Release:"4.12", FromRelease:"4.11", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m14s, firing for 0s:
Mar 28 01:58:02.976 - 74s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772593622743519232junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 36m28s on platformidentification.JobType{Release:"4.12", FromRelease:"4.11", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 36m28s, firing for 0s:
Mar 26 12:37:11.471 - 44s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 12:37:11.471 - 2144s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-altinfra-e2e-aws-ovn-localzones (all) - 10 runs, 70% failed, 57% of failures match = 40% impact
#1773168856378904576junit10 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773168856378904576junit10 hours ago
        <*errors.errorString | 0xc0012cf160>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.9:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-bcb969f5-fgr97\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711595315.19,\n      \"1\"\n    ]\n  }\n]",
        },
#1773168856378904576junit10 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773168856378904576junit10 hours ago
        <*errors.errorString | 0xc0016ad980>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.9:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-bcb969f5-fgr97\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711598119.653,\n      \"1\"\n    ]\n  }\n]",
        },
#1773075398561632256junit17 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773075398561632256junit17 hours ago
        <*errors.errorString | 0xc001bf0840>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.9:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7d4556c598-km4wc\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711572533.323,\n      \"1\"\n    ]\n  }\n]",
        },
#1773075398561632256junit17 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773075398561632256junit17 hours ago
        <*errors.errorString | 0xc000575d50>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithNoRunningPhase\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-in5qw7bl-c595f-9z2st-edge-us-west-2-lax-1a-zzb5m\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711575520.606,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithoutValidNode\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-in5qw7bl-c595f-9z2st-edge-us-west-2-lax-1a-zzb5m\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711575520.606,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.9:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7d4556c598-km4wc\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711575520.606,\n      \"1\"\n    ]\n  }\n]",
        },
#1772724225933578240junit39 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772724225933578240junit39 hours ago
        <*errors.errorString | 0xc001798160>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.21:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-bc6fb546b-6t49j\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711492338.396,\n      \"1\"\n    ]\n  }\n]",
        },
#1772724225933578240junit39 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772724225933578240junit39 hours ago
        <*errors.errorString | 0xc0022637a0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithNoRunningPhase\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-9t01nblv-c595f-9zltt-edge-us-east-1-chi-2a-2zdcb\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711495581.307,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithoutValidNode\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-9t01nblv-c595f-9zltt-edge-us-east-1-chi-2a-2zdcb\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711495581.307,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.21:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-bc6fb546b-6t49j\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711495581.307,\n      \"1\"\n    ]\n  }\n]",
        },
#1772667058400530432junit43 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772667058400530432junit43 hours ago
        <*errors.errorString | 0xc001b098a0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.26:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-574b9ccd7-76xtp\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711475846.958,\n      \"1\"\n    ]\n  }\n]",
        },
#1772667058400530432junit43 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772667058400530432junit43 hours ago
        <*errors.errorString | 0xc000e23aa0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithNoRunningPhase\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-wit2920d-c595f-dd27b-edge-us-west-2-lax-1b-g74lm\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711479269.199,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithoutValidNode\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-wit2920d-c595f-dd27b-edge-us-west-2-lax-1b-g74lm\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711479269.199,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.26:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-574b9ccd7-76xtp\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711479269.199,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-release-master-nightly-4.14-e2e-azure-ovn-cgroupsv1 (all) - 2 runs, 0% failed, 50% of runs match
#1773164525831327744junit10 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m38s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m38s, firing for 0s:
Mar 28 02:49:20.872 - 338s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-network-migration (all) - 9 runs, 33% failed, 100% of failures match = 33% impact
#1773157506583891968junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m42s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m42s, firing for 0s:
Mar 28 02:34:03.550 - 162s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773045681833906176junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m16s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 8m16s, firing for 0s:
Mar 27 19:09:30.288 - 496s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772684224491425792junit42 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m2s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 8m2s, firing for 0s:
Mar 26 19:13:09.694 - 482s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-ovn-serial (all) - 5 runs, 0% failed, 40% of runs match
#1773150717922512896junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 01:52:15.940 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772730461139767296junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 26 22:04:51.337 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-e2e-aws-sdn-serial (all) - 1 runs, 0% failed, 100% of runs match
#1773156167447482368junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m8s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=2s): pending for 1m8s, firing for 0s:
Mar 28 01:48:31.518 - 10s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:50:13.518 - 58s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-telco5g (all) - 6 runs, 17% failed, 300% of failures match = 50% impact
#1773150742241087488junit11 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1773150742241087488junit11 hours ago
        <*errors.errorString | 0xc0022cf770>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CannotRetrieveUpdates\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711592769.54,\n      \"1\"\n    ]\n  }\n]",
        },
#1773150742241087488junit11 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1773150742241087488junit11 hours ago
        <*errors.errorString | 0xc00222aa20>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CannotRetrieveUpdates\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711595381.434,\n      \"1\"\n    ]\n  }\n]",
        },
#1773058360795467776junit17 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1773058360795467776junit17 hours ago
        <*errors.errorString | 0xc000b022e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CannotRetrieveUpdates\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570868.335,\n      \"1\"\n    ]\n  }\n]",
        },
#1773058360795467776junit17 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1773058360795467776junit17 hours ago
        <*errors.errorString | 0xc001afa010>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CannotRetrieveUpdates\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711573430.11,\n      \"1\"\n    ]\n  }\n]",
        },
#1772619050929623040junit46 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1772619050929623040junit46 hours ago
        <*errors.errorString | 0xc000c6da10>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CannotRetrieveUpdates\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711466181.964,\n      \"1\"\n    ]\n  }\n]",
        },
#1772619050929623040junit46 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1772619050929623040junit46 hours ago
        <*errors.errorString | 0xc0023a25e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CannotRetrieveUpdates\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711468862.679,\n      \"1\"\n    ]\n  }\n]",
        },
pull-ci-openshift-origin-master-e2e-aws-ovn-cgroupsv2 (all) - 13 runs, 23% failed, 33% of failures match = 8% impact
#1773164761412800512junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 13m24s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 13m24s, firing for 0s:
Mar 28 02:34:53.098 - 134s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 28 02:34:53.098 - 186s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 02:34:53.098 - 456s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 02:44:01.098 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-serial (all) - 7 runs, 14% failed, 200% of failures match = 29% impact
#1773150709169000448junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 37m30s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 37m2s, firing for 28s:
Mar 28 01:31:17.492 - 22s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 01:31:17.492 - 22s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:32:07.492 - 174s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 01:32:07.492 - 2004s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:31:39.492 - 28s   E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772730499047886848junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m38s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m38s, firing for 0s:
Mar 26 21:37:32.223 - 458s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-sdn-techpreview (all) - 5 runs, 20% failed, 300% of failures match = 60% impact
#1773150717087846400junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 28 02:03:17.291 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772730464990138368junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 26 22:01:22.339 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619073587253248junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m28s, firing for 0s:
Mar 26 14:34:37.674 - 148s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619073587253248junit47 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|TechPreviewNoUpgrade|ClusterNotUpgradeable",alertstate="firing",severity!="info"} >= 1
    [
#1772619073587253248junit47 hours ago
        <*errors.errorString | 0xc000f54b20>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|TechPreviewNoUpgrade|ClusterNotUpgradeable\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1beta1.metrics.k8s.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711463749.15,\n      \"1\"\n    ]\n  }\n]",
        },
#1772619073587253248junit47 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|TechPreviewNoUpgrade|ClusterNotUpgradeable",alertstate="firing",severity!="info"} >= 1
    [
periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-sdn-upgrade (all) - 7 runs, 0% failed, 14% of runs match
#1773138649878106112junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 27m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 27m28s, firing for 0s:
Mar 28 00:51:37.524 - 1648s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-ovn-techpreview-serial (all) - 5 runs, 20% failed, 200% of failures match = 40% impact
#1773150720422318080junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 28 01:41:00.871 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619005849243648junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m58s, firing for 0s:
Mar 26 14:37:17.928 - 358s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.12-upgrade-from-nightly-4.11-ocp-ovn-remote-libvirt-s390x (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773153243497500672junit11 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.12", FromRelease:"4.11", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 28 02:42:40.342 - 28s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ibmcloud-ovn-multi-ppc64le (all) - 2 runs, 0% failed, 100% of runs match
#1773128034589609984junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m8s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 8m8s, firing for 0s:
Mar 28 01:12:43.002 - 488s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772765599173185536junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 34m2s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 34m2s, firing for 0s:
Mar 27 01:21:05.478 - 466s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 01:21:05.478 - 1576s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn (all) - 5 runs, 0% failed, 20% of runs match
#1773150722100039680junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m2s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m2s, firing for 0s:
Mar 28 01:28:01.112 - 182s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-ovn-kubernetes-master-e2e-ibmcloud-ipi-ovn-periodic (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
#1773138201620254720junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 43m42s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 43m42s, firing for 0s:
Mar 28 01:12:28.152 - 1786s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:13:46.152 - 508s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 01:14:06.152 - 328s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772775786315517952junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 29m24s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 29m24s, firing for 0s:
Mar 27 01:25:49.303 - 418s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 01:25:49.303 - 1048s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 01:26:25.303 - 298s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-ovn-kubernetes-release-4.14-e2e-ibmcloud-ipi-ovn-periodic (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
#1773138202463309824junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 22m54s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 22m54s, firing for 0s:
Mar 28 01:10:01.213 - 808s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 28 01:10:31.213 - 328s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 01:10:47.213 - 238s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772775790077808640junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 25m24s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 25m24s, firing for 0s:
Mar 27 01:19:19.836 - 418s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 01:19:19.836 - 838s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 01:20:17.836 - 268s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-agent-ha-dualstack-conformance (all) - 7 runs, 71% failed, 80% of failures match = 57% impact
#1773138212563193856junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m40s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m40s, firing for 0s:
Mar 28 01:18:21.113 - 160s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967201788137472junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h15m44s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h15m44s, firing for 0s:
Mar 27 14:04:15.608 - 3828s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 14:06:25.608 - 538s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 14:06:47.608 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772775803109511168junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m16s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 30m16s, firing for 0s:
Mar 27 01:28:53.692 - 1060s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 01:31:35.692 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 02:21:05.692 - 638s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772594529212305408junit2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m10s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m10s, firing for 0s:
Mar 26 13:32:18.202 - 550s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-serial-aws-ovn-heterogeneous (all) - 8 runs, 0% failed, 25% of runs match
#1773132144546156544junit12 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 53m48s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 53m48s, firing for 0s:
Mar 28 00:25:21.927 - 324s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 28 00:25:21.927 - 2904s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773058492366589952junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m4s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m4s, firing for 0s:
Mar 27 19:28:25.645 - 184s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-altinfra-e2e-vsphere-capi-zones (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
#1773137951941726208junit12 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773137951941726208junit12 hours ago
        <*errors.errorString | 0xc0024ea2c0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.18:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-5755f69b54-rnp88\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711587854.409,\n      \"1\"\n    ]\n  }\n]",
        },
#1773137951941726208junit12 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773137951941726208junit12 hours ago
        <*errors.errorString | 0xc00260a090>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.18:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-5755f69b54-rnp88\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711590599.592,\n      \"1\"\n    ]\n  }\n]",
        },
#1772775562402598912junit36 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772775562402598912junit36 hours ago
        <*errors.errorString | 0xc0009e27e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.24:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-645b5c745f-sdp2m\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711501741.907,\n      \"1\"\n    ]\n  }\n]",
        },
#1772775562402598912junit36 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772775562402598912junit36 hours ago
        <*errors.errorString | 0xc000a471b0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.24:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-645b5c745f-sdp2m\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711504232.815,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-aws-ovn-upgrade (all) - 10 runs, 50% failed, 40% of failures match = 20% impact
#1773120901403906048junit13 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 23:27:54.753 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773075606716551168junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m14s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m14s, firing for 0s:
Mar 27 20:31:35.687 - 74s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-cloud-credential-operator-release-4.14-periodics-e2e-azure-manual-oidc (all) - 2 runs, 0% failed, 50% of runs match
#1773119981542707200junit13 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 28 00:10:04.326 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-gcp-ovn-arm64 (all) - 6 runs, 0% failed, 33% of runs match
#1773131853616648192junit13 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m22s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"gcp", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m22s, firing for 0s:
Mar 28 00:12:05.288 - 142s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772632551370788864junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"gcp", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 26 15:11:42.231 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-aws-ovn-shared-vpc-edge-zones (all) - 8 runs, 0% failed, 13% of runs match
#1773113435823804416junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m26s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m26s, firing for 0s:
Mar 27 23:32:40.920 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 23:34:40.920 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-aws-ovn (all) - 64 runs, 8% failed, 80% of failures match = 6% impact
#1773113430786445312junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10m28s, firing for 0s:
Mar 27 23:28:32.187 - 120s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 23:28:32.187 - 450s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 23:37:04.187 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773108418500890624junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m4s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m4s, firing for 0s:
Mar 27 23:18:26.755 - 168s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 23:18:26.755 - 228s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 23:23:46.755 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772721592996990976junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m30s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m30s, firing for 0s:
Mar 26 21:28:13.081 - 150s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772674890478915584junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m54s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10m54s, firing for 0s:
Mar 26 18:30:55.978 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 18:30:55.978 - 298s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 18:37:25.978 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-ovn-workers-rhel8 (all) - 2 runs, 0% failed, 50% of runs match
#1773116962306527232junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m12s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m12s, firing for 0s:
Mar 27 23:36:34.220 - 254s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 23:42:20.220 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-altinfra-e2e-aws-ovn (all) - 10 runs, 60% failed, 67% of failures match = 40% impact
#1773113429985333248junit14 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773113429985333248junit14 hours ago
        <*errors.errorString | 0xc001a670a0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.8:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-569bc498bd-zp6ws\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711581746.092,\n      \"1\"\n    ]\n  }\n]",
        },
#1773113429985333248junit14 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773113429985333248junit14 hours ago
        <*errors.errorString | 0xc001b6e7d0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.8:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-569bc498bd-zp6ws\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711585216.67,\n      \"1\"\n    ]\n  }\n]",
        },
#1772994598453383168junit22 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772994598453383168junit22 hours ago
        <*errors.errorString | 0xc000c66dd0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.22:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6d667f75bd-nvkfb\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711554163.897,\n      \"1\"\n    ]\n  }\n]",
        },
#1772994598453383168junit22 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772994598453383168junit22 hours ago
        <*errors.errorString | 0xc0022add00>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.22:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6d667f75bd-nvkfb\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711557060.525,\n      \"1\"\n    ]\n  }\n]",
        },
#1772955098125701120junit25 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772955098125701120junit25 hours ago
        <*errors.errorString | 0xc002679500>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.16:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-67799bff97-jc9qk\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711543947.851,\n      \"1\"\n    ]\n  }\n]",
        },
#1772955098125701120junit25 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772955098125701120junit25 hours ago
        <*errors.errorString | 0xc0021be0e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.16:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-67799bff97-jc9qk\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711546784.962,\n      \"1\"\n    ]\n  }\n]",
        },
#1772861481688240128junit31 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772861481688240128junit31 hours ago
        <*errors.errorString | 0xc000d009a0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.34:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6f4bd5f5d-cj4h9\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711522183.713,\n      \"1\"\n    ]\n  }\n]",
        },
#1772861481688240128junit31 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772861481688240128junit31 hours ago
        <*errors.errorString | 0xc001c2dd10>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.34:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6f4bd5f5d-cj4h9\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711524901.121,\n      \"1\"\n    ]\n  }\n]",
        },
pull-ci-openshift-installer-master-altinfra-e2e-aws-custom-security-groups (all) - 9 runs, 56% failed, 100% of failures match = 56% impact
#1773113429943390208junit14 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773113429943390208junit14 hours ago
        <*errors.errorString | 0xc001c9a9d0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.29:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-569bc498bd-gsp6m\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711581957.319,\n      \"1\"\n    ]\n  }\n]",
        },
#1773113429943390208junit14 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773113429943390208junit14 hours ago
        <*errors.errorString | 0xc000ac0190>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.29:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-569bc498bd-gsp6m\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711585053.228,\n      \"1\"\n    ]\n  }\n]",
        },
#1773075398465163264junit17 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773075398465163264junit17 hours ago
        <*errors.errorString | 0xc002665900>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.27:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7d4556c598-6plw7\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711572875.457,\n      \"1\"\n    ]\n  }\n]",
        },
#1773075398465163264junit17 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773075398465163264junit17 hours ago
        <*errors.errorString | 0xc0007aaf60>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.27:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7d4556c598-6plw7\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711575594.027,\n      \"1\"\n    ]\n  }\n]",
        },
#1772994598403051520junit22 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772994598403051520junit22 hours ago
        <*errors.errorString | 0xc00156e820>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.28:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6d667f75bd-k5vmr\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711553705.781,\n      \"1\"\n    ]\n  }\n]",
        },
#1772994598403051520junit22 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772994598403051520junit22 hours ago
        <*errors.errorString | 0xc00090c9e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.28:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6d667f75bd-k5vmr\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711556841.379,\n      \"1\"\n    ]\n  }\n]",
        },
#1772724225849692160junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m54s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m54s, firing for 0s:
Mar 26 21:45:26.272 - 294s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772667058295672832junit43 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772667058295672832junit43 hours ago
        <*errors.errorString | 0xc001573d90>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.22:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-574b9ccd7-tfgjx\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711476088.867,\n      \"1\"\n    ]\n  }\n]",
        },
#1772667058295672832junit43 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772667058295672832junit43 hours ago
        <*errors.errorString | 0xc000570f40>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.22:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-574b9ccd7-tfgjx\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711479045.126,\n      \"1\"\n    ]\n  }\n]",
        },
pull-ci-openshift-installer-master-altinfra-e2e-aws-ovn-shared-vpc-edge-zones (all) - 9 runs, 56% failed, 100% of failures match = 56% impact
#1773113430069219328junit14 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773113430069219328junit14 hours ago
        <*errors.errorString | 0xc000e4b470>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.36:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-569bc498bd-vfgtr\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711581963.713,\n      \"1\"\n    ]\n  }\n]",
        },
#1773113430069219328junit14 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773113430069219328junit14 hours ago
        <*errors.errorString | 0xc001c80310>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.36:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-569bc498bd-vfgtr\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711584845.591,\n      \"1\"\n    ]\n  }\n]",
        },
#1773075398586798080junit16 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773075398586798080junit16 hours ago
        <*errors.errorString | 0xc000373150>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.8:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7d4556c598-2mddz\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711573043.775,\n      \"1\"\n    ]\n  }\n]",
        },
#1773075398586798080junit16 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773075398586798080junit16 hours ago
        <*errors.errorString | 0xc000ea9090>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.8:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7d4556c598-2mddz\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711576091.605,\n      \"1\"\n    ]\n  }\n]",
        },
#1772994598608572416junit22 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772994598608572416junit22 hours ago
        <*errors.errorString | 0xc001ea2a00>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.30:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6d667f75bd-l68qz\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711553735.914,\n      \"1\"\n    ]\n  }\n]",
        },
#1772994598608572416junit22 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772994598608572416junit22 hours ago
        <*errors.errorString | 0xc001306780>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.30:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6d667f75bd-l68qz\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711557069.881,\n      \"1\"\n    ]\n  }\n]",
        },
#1772861481780514816junit30 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772861481780514816junit30 hours ago
        <*errors.errorString | 0xc001775f10>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.19:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6f4bd5f5d-pmbj7\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711522318.803,\n      \"1\"\n    ]\n  }\n]",
        },
#1772861481780514816junit30 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772861481780514816junit30 hours ago
        <*errors.errorString | 0xc00177cd00>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.19:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6f4bd5f5d-pmbj7\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711525526.784,\n      \"1\"\n    ]\n  }\n]",
        },
#1772724225979715584junit39 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772724225979715584junit39 hours ago
        <*errors.errorString | 0xc001d8ed40>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.13:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-5b884476b4-kpsxn\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711492122.846,\n      \"1\"\n    ]\n  }\n]",
        },
#1772724225979715584junit39 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772724225979715584junit39 hours ago
        <*errors.errorString | 0xc001a950e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.13:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-5b884476b4-kpsxn\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711495117.876,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-upgrade (all) - 20 runs, 35% failed, 171% of failures match = 60% impact
#1773093689959124992junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m56s, firing for 0s:
Mar 27 22:01:43.218 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:07:13.218 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773093692484096000junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m26s, firing for 0s:
Mar 27 22:05:09.385 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:32:09.385 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773093687455125504junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m44s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m44s, firing for 0s:
Mar 27 23:07:22.760 - 166s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 23:11:40.760 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773093687455125504junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h1m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h1m54s, firing for 0s:
#1773093688289792000junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 15m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 15m54s, firing for 0s:
Mar 27 22:28:15.369 - 628s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:44:45.369 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:49:45.369 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773093689149624320junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10m56s, firing for 0s:
Mar 27 21:59:09.056 - 628s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:11:09.056 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967220297601024junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m26s, firing for 0s:
Mar 27 13:55:55.892 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 14:21:55.892 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967221950156800junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m56s, firing for 0s:
Mar 27 13:47:54.290 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 14:34:54.290 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967224424796160junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m26s, firing for 0s:
Mar 27 13:46:20.535 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:54:20.535 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967227373391872junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 13:43:51.768 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967222755463168junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 13:46:58.031 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967227817988096junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m28s, firing for 0s:
Mar 27 13:46:25.022 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967221081935872junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 13:37:30.857 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-azure-sdn-upgrade (all) - 2 runs, 0% failed, 50% of runs match
#1773093613631180800junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 22:31:47.474 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-altinfra-e2e-aws-ovn-fips (all) - 9 runs, 56% failed, 80% of failures match = 44% impact
#1773113430006304768junit14 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773113430006304768junit14 hours ago
        <*errors.errorString | 0xc00244ddf0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.28:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-569bc498bd-c9brd\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711581784.083,\n      \"1\"\n    ]\n  }\n]",
        },
#1773113430006304768junit14 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773113430006304768junit14 hours ago
        <*errors.errorString | 0xc002022f30>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.28:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-569bc498bd-c9brd\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711584554.075,\n      \"1\"\n    ]\n  }\n]",
        },
#1773075398523883520junit16 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773075398523883520junit16 hours ago
        <*errors.errorString | 0xc001847530>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.22:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7d4556c598-dnspn\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711572621.229,\n      \"1\"\n    ]\n  }\n]",
        },
#1773075398523883520junit16 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773075398523883520junit16 hours ago
        <*errors.errorString | 0xc001540a70>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.130.0.22:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-7d4556c598-dnspn\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711575880.845,\n      \"1\"\n    ]\n  }\n]",
        },
#1772994598503714816junit22 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772994598503714816junit22 hours ago
        <*errors.errorString | 0xc00246d630>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.27:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6d667f75bd-mq572\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711553480.156,\n      \"1\"\n    ]\n  }\n]",
        },
#1772994598503714816junit22 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772994598503714816junit22 hours ago
        <*errors.errorString | 0xc000ca9b00>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.27:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6d667f75bd-mq572\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711556490.42,\n      \"1\"\n    ]\n  }\n]",
        },
#1772667058366976000junit43 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772667058366976000junit43 hours ago
        <*errors.errorString | 0xc00257fcd0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.18:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-574b9ccd7-b5btq\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711475926.909,\n      \"1\"\n    ]\n  }\n]",
        },
#1772667058366976000junit43 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772667058366976000junit43 hours ago
        <*errors.errorString | 0xc000fd5890>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.18:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-574b9ccd7-b5btq\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711479005.582,\n      \"1\"\n    ]\n  }\n]",
        },
pull-ci-openshift-installer-master-e2e-vsphere-upi (all) - 5 runs, 60% failed, 133% of failures match = 80% impact
#1773108418668662784junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 23:03:37.333 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773012221484339200junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m26s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m26s, firing for 0s:
Mar 27 16:39:43.340 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 16:39:43.340 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772970599170183168junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 13:54:44.961 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772631661997985792junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 26 15:27:03.028 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-azurestack (all) - 12 runs, 42% failed, 40% of failures match = 17% impact
#1773097843783897088junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m54s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 12m54s, firing for 0s:
Mar 27 22:34:14.336 - 508s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:34:44.336 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 22:35:02.336 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772751629095800832junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m26s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m26s, firing for 0s:
Mar 26 23:56:58.316 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:57:58.316 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-azure-sdn-upgrade-out-of-change (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
#1773093680748433408junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m6s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m6s, firing for 0s:
Mar 27 22:57:22.236 - 186s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773093680748433408junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 43m30s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 43m30s, firing for 0s:
Mar 27 22:05:25.222 - 2462s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:50:29.222 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967180820811776junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 13:40:40.985 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-upgrade-out-of-change (all) - 2 runs, 0% failed, 100% of runs match
#1773093696674205696junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m28s, firing for 0s:
Mar 27 22:01:54.247 - 388s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967216875048960junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 13:46:06.517 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-techpreview-serial (all) - 2 runs, 0% failed, 100% of runs match
#1773093597965455360junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 22:02:39.621 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967245413093376junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m58s, firing for 0s:
Mar 27 13:36:51.688 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-azure-ovn (all) - 11 runs, 0% failed, 9% of runs match
#1773086528017272832junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 22:33:37.579 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-serial (all) - 2 runs, 0% failed, 50% of runs match
#1773093652264914944junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m58s, firing for 0s:
Mar 27 22:04:26.601 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-upgrade-from-stable-4.15-ocp-e2e-upgrade-azure-ovn-arm64 (all) - 3 runs, 33% failed, 300% of failures match = 100% impact
#1773077372229128192junit14 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 39m14s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 39m14s, firing for 0s:
Mar 27 22:55:28.362 - 2354s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773023571891720192junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m48s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m48s, firing for 0s:
Mar 27 18:33:00.611 - 168s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773023571891720192junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 35m44s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 35m44s, firing for 0s:
Mar 27 17:52:21.555 - 2144s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967128907911168junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 13m52s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 13m52s, firing for 0s:
Mar 27 14:09:34.888 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 14:16:34.888 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 14:26:04.888 - 388s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 14:37:04.888 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-machine-config-operator-master-e2e-azure-ovn-upgrade-out-of-change (all) - 15 runs, 20% failed, 233% of failures match = 47% impact
#1773084826044207104junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 21:37:37.052 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772995101732114432junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 16m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 16m56s, firing for 0s:
Mar 27 15:40:19.821 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 16:05:19.821 - 568s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772863379363336192junit28 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m58s, firing for 0s:
Mar 27 06:58:30.752 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772776475028623360junit35 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 13m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 13m56s, firing for 0s:
Mar 27 01:05:21.836 - 748s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 01:30:21.836 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772758877067022336junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 00:04:01.938 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772743756911480832junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m26s, firing for 0s:
Mar 26 23:05:52.724 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:10:22.724 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772638183658360832junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 11m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 11m26s, firing for 0s:
Mar 26 16:04:11.209 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 16:29:11.209 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
rehearse-49787-pull-ci-openshift-origin-release-4.14-e2e-aws-csi (all) - 1 runs, 0% failed, 100% of runs match
#1773115317094977536junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 52s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 52s, firing for 0s:
Mar 27 23:21:33.170 - 52s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-sdn-upgrade (all) - 20 runs, 5% failed, 300% of failures match = 15% impact
#1773093672762478592junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 18m50s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 18m50s, firing for 0s:
Mar 27 21:45:55.434 - 894s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:02:21.434 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:15:21.434 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967235334180864junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 24m42s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 24m42s, firing for 0s:
Mar 27 13:27:33.981 - 1482s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967234491125760junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 50m22s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 50m22s, firing for 0s:
Mar 27 13:22:02.986 - 432s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 13:22:02.986 - 2562s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 14:07:16.986 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-sdn-upgrade (all) - 8 runs, 38% failed, 33% of failures match = 13% impact
#1773096904620511232junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 15m46s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 15m46s, firing for 0s:
Mar 27 21:54:59.556 - 946s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade (all) - 15 runs, 33% failed, 20% of failures match = 7% impact
#1773093093868834816junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m34s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28m34s, firing for 0s:
Mar 27 22:17:40.280 - 1714s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-4.16-upgrade-from-stable-4.15-e2e-azure-ovn-upgrade (all) - 7 runs, 29% failed, 250% of failures match = 71% impact
#1773071911274680320junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h3m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h3m28s, firing for 0s:
Mar 27 21:04:47.177 - 3808s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773045681590636544junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m50s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m50s, firing for 0s:
Mar 27 20:12:33.117 - 350s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773045681590636544junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 39m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 39m28s, firing for 0s:
Mar 27 19:10:55.271 - 628s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:37:25.271 - 958s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:54:25.271 - 782s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772936019478319104junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m14s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m14s, firing for 0s:
Mar 27 12:42:56.351 - 134s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772936019478319104junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 33m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 33m54s, firing for 0s:
Mar 27 12:03:41.848 - 2034s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772714058068267008junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 34m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 34m28s, firing for 0s:
Mar 26 21:32:39.534 - 2068s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772714058068267008junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m2s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m2s, firing for 0s:
Mar 26 22:12:04.820 - 182s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772591487603707904junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2h9m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2h9m56s, firing for 0s:
Mar 26 13:05:10.837 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 13:05:10.837 - 7618s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-techpreview (all) - 2 runs, 0% failed, 100% of runs match
#1773093642152448000junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 22:06:14.321 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967191709224960junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 13:41:07.690 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn-upgrade (all) - 8 runs, 25% failed, 50% of failures match = 13% impact
#1773071916932796416junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 32m22s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 32m22s, firing for 0s:
Mar 27 20:31:25.264 - 86s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 20:31:25.264 - 1856s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-vsphere-ovn-techpreview-serial (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1773093623726870528junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 21:55:31.564 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn (all) - 2 runs, 0% failed, 100% of runs match
#1773093641737211904junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 22:05:58.815 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967185874948096junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 13:47:57.444 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-api-master-e2e-aws-ovn (all) - 25 runs, 16% failed, 25% of failures match = 4% impact
#1773096730305236992junit15 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 15m56s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 15m56s, firing for 0s:
Mar 27 22:10:16.767 - 68s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 27 22:10:16.767 - 250s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 22:10:16.767 - 490s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 22:19:58.767 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-oc-master-e2e-aws-ovn-serial (all) - 7 runs, 43% failed, 33% of failures match = 14% impact
#1773081350610358272junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m38s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m38s, firing for 0s:
Mar 27 21:06:11.900 - 278s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-azure-csi (all) - 2 runs, 50% failed, 200% of failures match = 100% impact
#1773093612830068736junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 22:08:29.249 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772967244645535744junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 13:41:44.622 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-csi-operator-master-e2e-azure-file-nfs-csi (all) - 3 runs, 0% failed, 67% of runs match
#1773094204231127040junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 22:21:38.393 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773026395539116032junit21 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m58s, firing for 0s:
Mar 27 17:36:00.224 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-vsphere-ovn-upi (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1773093624561537024junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m44s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m44s, firing for 0s:
Mar 27 21:38:55.476 - 172s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 21:38:55.476 - 412s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773093624561537024junit16 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1773093624561537024junit16 hours ago
        <*errors.errorString | 0xc001cf1430>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1beta1.metrics.k8s.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711575616.097,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-release-master-nightly-4.14-e2e-metal-ipi-sdn-serial-ipv4 (all) - 5 runs, 40% failed, 50% of failures match = 20% impact
#1773058324711870464junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2h32m52s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2h32m52s, firing for 0s:
Mar 27 20:12:44.095 - 7616s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:13:14.095 - 1168s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 20:14:34.095 - 388s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-nightly-4.12-ocp-ovn-remote-libvirt-s390x (all) - 4 runs, 25% failed, 200% of failures match = 50% impact
#1773077683412930560junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m28s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m28s, firing for 0s:
Mar 27 21:50:03.229 - 328s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772896494706233344junit28 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 09:58:39.686 - 58s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-csi (all) - 2 runs, 0% failed, 50% of runs match
#1773093616982429696junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 18m24s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 18m24s, firing for 0s:
Mar 27 21:46:45.811 - 348s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 21:46:45.811 - 756s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-upgrade-aws-ovn-arm64 (all) - 3 runs, 0% failed, 33% of runs match
#1773077373760049152junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 26m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 26m56s, firing for 0s:
Mar 27 20:39:13.955 - 170s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 20:39:13.955 - 1446s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773077373760049152junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m22s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m22s, firing for 0s:
pull-ci-openshift-ovn-kubernetes-master-e2e-azure-ovn-upgrade (all) - 3 runs, 67% failed, 100% of failures match = 67% impact
#1773058166032961536junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 20:21:28.097 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772716776312803328junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 13m24s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 13m24s, firing for 0s:
Mar 26 21:13:10.342 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 21:38:10.342 - 328s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 21:44:10.342 - 388s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-upgrade-from-nightly-4.15-ocp-e2e-upgrade-gcp-ovn-arm64 (all) - 1 runs, 0% failed, 100% of runs match
#1773065578668363776junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m22s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"gcp", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 8m22s, firing for 0s:
Mar 27 20:16:54.423 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:21:24.423 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:23:24.423 - 178s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:26:54.423 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-master-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade (all) - 4 runs, 50% failed, 100% of failures match = 50% impact
#1773058158491602944junit16 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 25m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 25m28s, firing for 0s:
Mar 27 19:39:05.774 - 1528s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772638381545623552junit44 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772638381545623552junit44 hours ago
        <*errors.errorString | 0xc001cdd090>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeDeploymentReplicasMismatch\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-main\",\n      \"deployment\": \"ovnkube-control-plane\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711472646.008,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"ovnkube-cluster-manager\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"pod\": \"ovnkube-control-plane-79fc89498d-dfs6d\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"92ddbe24-6af4-4fec-8a09-826d2dd7474d\"\n    },\n    \"value\": [\n      1711472646.008,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"ovnkube-cluster-manager\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"pod\": \"ovnkube-control-plane-79fc89498d-pds52\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"0fe0aeab-5b87-4562-99ed-7b387561166f\"\n    },\n    \"value\": [\n      1711472646.008,\n      \"1\"\n    ]\n  }\n]",
        },
#1772638381545623552junit44 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772638381545623552junit44 hours ago
        <*errors.errorString | 0xc0015a27b0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"ClusterOperatorDegraded\",\n      \"alertstate\": \"firing\",\n      \"name\": \"network\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"RolloutHung\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711475706.834,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeDeploymentReplicasMismatch\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-main\",\n      \"deployment\": \"ovnkube-control-plane\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711475706.834,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"ovnkube-cluster-manager\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"pod\": \"ovnkube-control-plane-79fc89498d-dfs6d\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"92ddbe24-6af4-4fec-8a09-826d2dd7474d\"\n    },\n    \"value\": [\n      1711475706.834,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"ovnkube-cluster-manager\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"pod\": \"ovnkube-control-plane-79fc89498d-pds52\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"0fe0aeab-5b87-4562-99ed-7b387561166f\"\n    },\n    \"value\": [\n      1711475706.834,\n      \"1\"\n    ]\n  }\n]",
        },
rehearse-50309-periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-ovn-remote-libvirt-s390x (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773075238192418816junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m56s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m56s, firing for 0s:
Mar 27 20:27:04.222 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:30:34.222 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-ibmcloud-ovn (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1773061830894161920junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30m42s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 30m42s, firing for 0s:
Mar 27 20:35:02.351 - 322s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 20:35:02.351 - 1432s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:36:46.351 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
pull-ci-openshift-cluster-baremetal-operator-master-e2e-metal-ipi-serial-ipv4 (all) - 5 runs, 80% failed, 25% of failures match = 20% impact
#1773041304230432768junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2h30m16s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2h30m16s, firing for 0s:
Mar 27 19:37:20.621 - 7160s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:38:20.621 - 1588s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 19:38:56.621 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-sdn-upgrade (all) - 5 runs, 0% failed, 20% of runs match
#1773058324896419840junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m26s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m26s, firing for 0s:
Mar 27 19:44:40.592 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:57:40.592 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-stable-4.14-ocp-e2e-upgrade-azure-ovn-arm64 (all) - 8 runs, 13% failed, 100% of failures match = 13% impact
#1773057570727006208junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 20:18:39.936 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-upgrade-from-stable-4.15-ocp-e2e-upgrade-azure-ovn-heterogeneous (all) - 5 runs, 20% failed, 200% of failures match = 40% impact
#1773046643663310848junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 20:42:33.114 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773021581610586112junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 26m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 26m54s, firing for 0s:
Mar 27 18:08:02.813 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 18:15:32.813 - 808s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 18:31:02.813 - 598s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-azure-sdn-techpreview-serial (all) - 5 runs, 80% failed, 50% of failures match = 40% impact
#1773058324799950848junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 19:46:18.738 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772730516148064256junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 26 22:03:48.240 - 88s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-operator-framework-olm-master-e2e-gcp-ovn (all) - 5 runs, 0% failed, 20% of runs match
#1773065040782430208junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m38s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m38s, firing for 0s:
Mar 27 20:16:45.516 - 34s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 20:16:45.516 - 364s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773065040782430208junit17 hours ago
        <*errors.errorString | 0xc0016bf900>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.apps.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.authorization.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.build.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.image.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.oauth.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.packages.operators.coreos.com\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.project.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.quota.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.route.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.security.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.template.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1.user.openshift.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711570746.088,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\...
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ovn-serial-aws-arm64 (all) - 6 runs, 33% failed, 50% of failures match = 17% impact
#1773057895357747200junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m52s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m52s, firing for 0s:
Mar 27 19:19:22.823 - 172s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-upgrade-gcp-ovn-heterogeneous (all) - 5 runs, 0% failed, 20% of runs match
#1773074640835448832junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m14s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m14s, firing for 0s:
Mar 27 20:41:05.509 - 74s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-stable-4.13-ocp-e2e-aws-ovn-heterogeneous-upgrade (all) - 7 runs, 57% failed, 25% of failures match = 14% impact
#1773058492525973504junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h45m44s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h45m44s, firing for 0s:
Mar 27 19:42:37.683 - 1310s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 19:42:37.683 - 5034s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-aws-ovn-heterogeneous-upgrade (all) - 5 runs, 0% failed, 20% of runs match
#1773074640818671616junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 19m12s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 19m12s, firing for 0s:
Mar 27 20:39:21.696 - 824s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 20:54:37.696 - 328s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-gcp-sdn-techpreview-serial (all) - 5 runs, 60% failed, 67% of failures match = 40% impact
#1773058335612866560junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 24m52s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 24m52s, firing for 0s:
Mar 27 19:22:06.224 - 146s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 19:22:06.224 - 1346s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772730500218097664junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 27m36s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 27m36s, firing for 0s:
Mar 26 21:41:58.356 - 154s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 21:41:58.356 - 1474s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 21:46:04.356 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
periodic-ci-openshift-release-master-nightly-4.14-upgrade-from-stable-4.13-e2e-metal-ipi-sdn-bm-upgrade (all) - 5 runs, 100% failed, 20% of failures match = 20% impact
#1773058345670807552junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"metal", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 20:56:55.208 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade (all) - 70 runs, 6% failed, 75% of failures match = 4% impact
#1773047443479334912junit17 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 38m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 38m58s, firing for 0s:
Mar 27 18:51:15.367 - 2338s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752591520796672junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m40s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m40s, firing for 0s:
Mar 27 00:28:07.040 - 280s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772752591520796672junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h22m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h22m54s, firing for 0s:
Mar 26 23:19:48.684 - 1288s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 23:19:48.684 - 2188s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:57:48.684 - 538s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 00:07:18.684 - 960s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772660980992446464junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 36m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 36m58s, firing for 0s:
Mar 26 17:11:16.342 - 2218s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-e2e-aws-sdn-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1773046704333918208junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m8s on platformidentification.JobType{Release:"4.12", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m8s, firing for 0s:
Mar 27 18:45:35.712 - 128s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-vsphere-ovn-serial (all) - 5 runs, 60% failed, 67% of failures match = 40% impact
#1773058351551221760junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m28s, firing for 0s:
Mar 27 19:33:49.831 - 328s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772619027441520640junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 26 14:37:37.748 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-csi-operator-master-e2e-azure-csi (all) - 3 runs, 33% failed, 200% of failures match = 67% impact
#1773067016576438272junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m28s, firing for 0s:
Mar 27 20:26:08.453 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1773026395476201472junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m28s, firing for 0s:
Mar 27 17:40:08.347 - 268s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-master-e2e-azure-ovn (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
#1773058165210877952junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 19:43:23.531 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-gcp-ovn (all) - 21 runs, 19% failed, 25% of failures match = 5% impact
#1773053733182115840junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m6s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m6s, firing for 0s:
Mar 27 19:30:00.464 - 368s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:30:10.464 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
pull-ci-openshift-origin-master-e2e-gcp-ovn-techpreview (all) - 3 runs, 33% failed, 100% of failures match = 33% impact
#1773053981883371520junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 11m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 11m28s, firing for 0s:
Mar 27 19:27:08.239 - 194s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 19:27:08.239 - 494s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-upgrade-gcp-ovn-heterogeneous (all) - 7 runs, 0% failed, 14% of runs match
#1773058492618248192junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m2s on platformidentification.JobType{Release:"4.14", FromRelease:"4.14", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m2s, firing for 0s:
Mar 27 19:26:29.177 - 62s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-gcp-csi (all) - 11 runs, 9% failed, 100% of failures match = 9% impact
#1773057790701473792junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 14m48s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 14m48s, firing for 0s:
Mar 27 19:51:33.403 - 888s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-stable-4.14-ocp-e2e-aws-ovn-heterogeneous-upgrade (all) - 8 runs, 13% failed, 100% of failures match = 13% impact
#1773051203328937984junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m52s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m52s, firing for 0s:
Mar 27 19:02:36.478 - 292s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-aws-ovn-arm64 (all) - 6 runs, 0% failed, 17% of runs match
#1773057897555562496junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m10s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m10s, firing for 0s:
Mar 27 19:18:05.163 - 102s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 19:21:19.163 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn (all) - 8 runs, 13% failed, 100% of failures match = 13% impact
#1773045686581858304junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m20s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m20s, firing for 0s:
Mar 27 18:45:08.083 - 140s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-ipsec-step-registry (all) - 6 runs, 33% failed, 100% of failures match = 33% impact
#1773045694890774528junit18 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m46s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m46s, firing for 0s:
Mar 27 18:49:05.039 - 8s    I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 18:49:05.039 - 458s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772936032346443776junit26 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 19m48s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 19m48s, firing for 0s:
Mar 27 11:44:06.253 - 234s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 11:44:06.253 - 954s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-release-4.13-4.13-upgrade-from-stable-4.12-e2e-aws-ovn-upgrade (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1773027126077820928junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 40s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 40s, firing for 0s:
Mar 27 17:31:15.266 - 40s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-release-4.13-4.13-upgrade-from-stable-4.12-local-gateway-e2e-aws-ovn-upgrade (all) - 3 runs, 0% failed, 33% of runs match
#1773027126161707008junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 39m44s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 39m44s, firing for 0s:
Mar 27 17:26:36.863 - 2384s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-sdn-serial (all) - 1 runs, 0% failed, 100% of runs match
#1773033869721735168junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m8s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m8s, firing for 0s:
Mar 27 17:46:15.585 - 188s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-monitoring-operator-master-e2e-aws-ovn-upgrade (all) - 12 runs, 17% failed, 100% of failures match = 17% impact
#1773040117867352064junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h24m50s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h20m52s, firing for 3m58s:
Mar 27 18:16:35.944 - 56s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 18:16:35.944 - 56s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 18:21:29.944 - 1386s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 18:21:29.944 - 3354s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 18:17:31.944 - 238s  E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772659313559474176junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h8m38s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h8m38s, firing for 0s:
Mar 26 17:03:40.189 - 30s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 26 17:03:40.189 - 440s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 17:03:40.189 - 3350s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:12:32.189 - 298s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-upgrade-local-gateway (all) - 3 runs, 0% failed, 33% of runs match
#1773027126895710208junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 23m54s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 23m54s, firing for 0s:
Mar 27 17:23:36.229 - 1196s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 17:45:04.229 - 238s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-upgrade (all) - 3 runs, 0% failed, 33% of runs match
#1773027126455308288junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 30s, firing for 0s:
Mar 27 17:20:58.177 - 30s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-azure-ovn-upi (all) - 4 runs, 75% failed, 67% of failures match = 50% impact
#1773012221379481600junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 17:48:08.139 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772631661872156672junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m56s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m56s, firing for 0s:
Mar 26 16:41:58.048 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 16:41:58.048 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-insights-operator-master-e2e-agnostic-upgrade (all) - 4 runs, 25% failed, 300% of failures match = 75% impact
#1773025693844639744junit19 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 18:04:48.974 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772991527711150080junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m26s, firing for 0s:
Mar 27 15:22:10.429 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 15:46:10.429 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772916399782498304junit27 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m54s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m54s, firing for 0s:
Mar 27 10:24:12.685 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 10:27:42.685 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 10:49:12.685 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-azure-ovn (all) - 3 runs, 0% failed, 33% of runs match
#1773027128556654592junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 17:33:08.912 - 58s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
release-openshift-origin-installer-e2e-aws-disruptive-4.14 (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
#1773030095343063040junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h2m40s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 40m14s, firing for 22m26s:
Mar 27 17:37:00.416 - 222s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 17:37:30.416 - 192s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 17:37:42.416 - 180s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 27 17:46:10.416 - 392s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 17:46:10.416 - 392s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 17:46:10.416 - 392s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 27 18:09:40.416 - 322s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 18:09:40.416 - 322s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 17:40:42.416 - 328s  E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
Mar 27 17:52:42.416 - 1018s E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
#1772667703337684992junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 40m38s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 32m32s, firing for 8m6s:
Mar 26 17:51:37.749 - 1006s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:52:37.749 - 946s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 18:08:23.749 - 486s  E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above info
KubeAPIErrorBudgetBurn was at or above info for at least 8m6s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 32m32s, firing for 8m6s:
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-local-to-shared-gateway-mode-migration (all) - 3 runs, 0% failed, 33% of runs match
#1773027126371422208junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m22s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m22s, firing for 0s:
Mar 27 17:24:21.229 - 82s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-ovn-hybrid-step-registry (all) - 3 runs, 33% failed, 100% of failures match = 33% impact
#1773027132839038976junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 17:20:34.354 - 58s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-local-gateway (all) - 3 runs, 0% failed, 33% of runs match
#1773027126329479168junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m32s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m32s, firing for 0s:
Mar 27 17:19:40.120 - 46s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 17:19:40.120 - 226s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-crun-wasm (all) - 2 runs, 0% failed, 50% of runs match
#1773023551536762880junit20 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m50s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m50s, firing for 0s:
Mar 27 17:04:10.260 - 70s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 17:04:10.260 - 400s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-ovn (all) - 3 runs, 33% failed, 100% of failures match = 33% impact
#1773017311737810944junit21 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 16:59:45.056 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-ovn-upgrade-out-of-change (all) - 3 runs, 33% failed, 100% of failures match = 33% impact
#1773017311817502720junit21 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m26s, firing for 0s:
Mar 27 17:01:22.920 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 17:25:52.920 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-release-4.15-4.15-upgrade-from-stable-4.14-e2e-azure-ovn-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772981757537161216junit21 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 14:44:37.343 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-aws-ovn-upgrade (all) - 16 runs, 56% failed, 11% of failures match = 6% impact
#1772993600615878656junit21 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 48m34s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 48m34s, firing for 0s:
Mar 27 15:13:56.332 - 422s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 15:13:56.332 - 2492s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-csi-operator-master-e2e-azure (all) - 3 runs, 0% failed, 33% of runs match
#1772996636553777152junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 15:47:50.052 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-api-master-e2e-azure (all) - 21 runs, 38% failed, 25% of failures match = 10% impact
#1772996908730552320junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 15:43:52.812 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772934955031072768junit26 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 11:41:18.786 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ovn-remote-libvirt-s390x (all) - 4 runs, 75% failed, 67% of failures match = 50% impact
#1773002182656790528junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m56s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m56s, firing for 0s:
Mar 27 15:31:58.759 - 118s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 15:34:28.759 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772821000090554368junit34 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m28s, firing for 0s:
Mar 27 03:33:08.022 - 208s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64-techpreview (all) - 4 runs, 0% failed, 25% of runs match
#1772986372173336576junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m2s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m2s, firing for 0s:
Mar 27 14:35:07.756 - 62s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-release-4.15-4.15-upgrade-from-stable-4.14-e2e-aws-ovn-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772981757507801088junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m34s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m34s, firing for 0s:
Mar 27 14:18:06.284 - 456s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 14:26:44.284 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-gcp-sdn-techpreview-serial (all) - 4 runs, 75% failed, 33% of failures match = 25% impact
#1772986574674333696junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 23m48s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 23m48s, firing for 0s:
Mar 27 14:38:12.215 - 204s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 14:38:12.215 - 1224s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-api-master-e2e-upgrade (all) - 28 runs, 21% failed, 17% of failures match = 4% impact
#1772991605909753856junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 51m12s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 51m12s, firing for 0s:
Mar 27 15:03:51.548 - 8s    I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 15:03:51.548 - 1538s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 15:05:31.548 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 15:31:01.548 - 1078s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-sdn-serial-aws-arm64 (all) - 4 runs, 0% failed, 25% of runs match
#1772986371955232768junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m26s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m26s, firing for 0s:
Mar 27 14:30:11.229 - 86s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-gcp-sdn-serial (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1772986520991436800junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m6s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m6s, firing for 0s:
Mar 27 14:32:18.441 - 186s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-sdn-upgrade (all) - 4 runs, 0% failed, 50% of runs match
#1772986534404820992junit22 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 17m18s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 17m18s, firing for 0s:
Mar 27 14:34:13.903 - 1038s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772707761855926272junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 24m36s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 24m36s, firing for 0s:
Mar 26 20:07:20.865 - 1476s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-ovn-remote-libvirt-s390x (all) - 4 runs, 0% failed, 50% of runs match
#1772987084517150720junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 37m24s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 37m24s, firing for 0s:
Mar 27 14:46:17.311 - 1978s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 15:32:17.311 - 208s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 15:36:47.311 - 58s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772805905839034368junit35 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m46s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10m46s, firing for 0s:
Mar 27 02:48:00.635 - 128s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 02:48:00.635 - 518s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-csi-operator-master-e2e-azure-file-csi (all) - 4 runs, 25% failed, 200% of failures match = 50% impact
#1772996636595720192junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 15:39:50.826 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772961436004257792junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m58s, firing for 0s:
Mar 27 13:24:31.563 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-gcp-ovn (all) - 4 runs, 0% failed, 25% of runs match
#1772986607498956800junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m28s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m28s, firing for 0s:
Mar 27 14:35:48.557 - 148s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-serial-aws-ovn-heterogeneous (all) - 6 runs, 33% failed, 50% of failures match = 17% impact
#1772967557674831872junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 52m38s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 52m38s, firing for 0s:
Mar 27 13:39:50.794 - 216s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 13:39:50.794 - 2436s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:44:58.794 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 14:21:58.794 - 448s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-upgrade-gcp-ovn-arm64 (all) - 3 runs, 33% failed, 100% of failures match = 33% impact
#1772967128828219392junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 36m48s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"gcp", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 36m48s, firing for 0s:
Mar 27 13:27:18.336 - 144s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 13:27:18.336 - 2064s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-serial (all) - 2 runs, 50% failed, 200% of failures match = 100% impact
#1772977749372178432junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m56s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 5m56s, firing for 0s:
Mar 27 13:59:51.569 - 356s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772615357794619392junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m40s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 7m40s, firing for 0s:
Mar 26 14:00:02.679 - 460s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-ovn-fips (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1772986584015048704junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 32s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 32s, firing for 0s:
Mar 27 14:35:38.866 - 32s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-aws-sdn-techpreview (all) - 4 runs, 0% failed, 25% of runs match
#1772986588205158400junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 22s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 22s, firing for 0s:
Mar 27 14:34:25.214 - 22s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-telco5g (all) - 4 runs, 0% failed, 25% of runs match
#1772967205135192064junit23 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772967205135192064junit23 hours ago
        <*errors.errorString | 0xc0017627e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CannotRetrieveUpdates\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711549245.297,\n      \"1\"\n    ]\n  }\n]",
        },
#1772967205135192064junit23 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772967205135192064junit23 hours ago
        <*errors.errorString | 0xc000b86090>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CannotRetrieveUpdates\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711552060.753,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-release-master-ci-4.13-e2e-aws-ovn (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1772986568814891008junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m18s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m18s, firing for 0s:
Mar 27 14:37:43.210 - 54s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 14:37:43.210 - 84s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-aws-ovn-upgrade (all) - 7 runs, 29% failed, 50% of failures match = 14% impact
#1772958855873957888junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m8s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m8s, firing for 0s:
Mar 27 12:47:47.175 - 68s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-sdn-arm64 (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1772986372290777088junit23 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m8s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m8s, firing for 0s:
Mar 27 14:34:00.521 - 68s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-vsphere-ovn-upi (all) - 4 runs, 50% failed, 100% of failures match = 50% impact
#1772986571331473408junit23 hours ago
        <*errors.errorString | 0xc001e75d10>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711549885.306,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711549885.306,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711549885.306,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711549885.306,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711549885.306,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711549885.306,\n      \"1\"\n    ]\n  }\n]",
        },
#1772986571331473408junit23 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1772986571331473408junit23 hours ago
        <*errors.errorString | 0xc001b27e60>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711552089.846,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711552089.846,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711552089.846,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711552089.846,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711552089.846,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711552089.846,\n      \"1\"\n    ]\n  }\n]",
        },
#1772986571331473408junit23 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1772617099869425664junit47 hours ago
        <*errors.errorString | 0xc0019f8170>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464714.84,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464714.84,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464714.84,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464714.84,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464714.84,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464714.84,\n      \"1\"\n    ]\n  }\n]",
        },
#1772617099869425664junit47 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1772617099869425664junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m52s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m52s, firing for 0s:
Mar 26 14:12:43.597 - 42s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 14:12:43.597 - 402s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 14:19:57.597 - 28s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-aws-sdn-upgrade (all) - 5 runs, 20% failed, 100% of failures match = 20% impact
#1772958856037535744junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m20s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m20s, firing for 0s:
Mar 27 12:42:35.109 - 140s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-vsphere-ovn-techpreview (all) - 2 runs, 0% failed, 50% of runs match
#1772967193403723776junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 13:40:16.817 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-vsphere-ovn-upi-serial (all) - 2 runs, 0% failed, 50% of runs match
#1772967194234195968junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 15m24s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 15m24s, firing for 0s:
Mar 27 13:19:30.035 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 13:19:30.035 - 838s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:35:00.035 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
rehearse-44057-periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-ovn-remote-libvirt-ppc64le (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
#1772974874231312384junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m30s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"libvirt", Architecture:"ppc64le", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m30s, firing for 0s:
Mar 27 13:46:44.665 - 90s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 13:46:44.665 - 240s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.16-e2e-vsphere-sdn (all) - 2 runs, 0% failed, 50% of runs match
#1772967259677921280junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 13:35:52.111 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.15-upgrade-from-stable-4.14-e2e-vsphere-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772961643236429824junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m26s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m26s, firing for 0s:
Mar 27 13:11:07.321 - 118s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 13:14:37.321 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-okd-4.15-upgrade-from-okd-4.14-e2e-aws-ovn-upgrade (all) - 1 runs, 0% failed, 100% of runs match
#1772961643441950720junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m16s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m16s, firing for 0s:
Mar 27 12:59:03.482 - 196s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-vsphere-ovn-upgrade-storage-data (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772961144449798144junit24 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m58s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10m58s, firing for 0s:
Mar 27 13:07:01.458 - 658s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-serial (all) - 6 runs, 33% failed, 50% of failures match = 17% impact
#1772936020254265344junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m46s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 12m46s, firing for 0s:
Mar 27 11:22:40.110 - 168s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:27:00.110 - 598s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn (all) - 18 runs, 22% failed, 25% of failures match = 6% impact
#1772950614108540928junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m26s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m26s, firing for 0s:
Mar 27 12:22:24.647 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 12:29:24.647 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-etcd-operator-master-e2e-agnostic-ovn (all) - 1 runs, 0% failed, 100% of runs match
#1772948176471330816junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m8s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10m8s, firing for 0s:
Mar 27 12:11:14.016 - 170s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 12:11:14.016 - 200s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 12:15:36.016 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-azure-ovn (all) - 6 runs, 17% failed, 200% of failures match = 33% impact
#1772936023571959808junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 11:53:05.538 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772714081057247232junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 26 21:04:11.400 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.16-upgrade-from-nightly-4.15-ocp-ovn-remote-libvirt-ppc64le (all) - 4 runs, 0% failed, 25% of runs match
#1772941783920021504junit25 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m26s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"libvirt", Architecture:"ppc64le", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m26s, firing for 0s:
Mar 27 12:47:59.709 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 12:53:59.709 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
rehearse-49815-pull-ci-openshift-cluster-api-provider-baremetal-release-4.16-e2e-metal-ipi (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772918276649652224junit26 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 55m50s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 55m50s, firing for 0s:
Mar 27 11:09:49.385 - 1558s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 11:10:49.385 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 11:46:19.385 - 1554s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-aws-ovn-imdsv2 (all) - 8 runs, 25% failed, 50% of failures match = 13% impact
#1772861485848989696junit30 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 24m30s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 21m8s, firing for 3m22s:
Mar 27 07:12:57.363 - 94s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 07:12:57.363 - 1174s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 07:09:35.363 - 202s  E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above info
KubeAPIErrorBudgetBurn was at or above info for at least 3m22s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 21m8s, firing for 3m22s:
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ovn-ppc64le-powervs (all) - 4 runs, 100% failed, 25% of failures match = 25% impact
#1772866295696461824junit29 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 51m54s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"", Architecture:"ppc64le", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 51m54s, firing for 0s:
Mar 27 07:32:49.055 - 2068s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 07:33:35.055 - 388s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 27 07:33:49.055 - 658s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1772866295696461824junit29 hours ago
# [sig-instrumentation][Late] OpenShift alerting rules [apigroup:image.openshift.io] should link to a valid URL if the runbook_url annotation is defined [Suite:openshift/conformance/parallel]
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:113]: Failed to fetch alerting rules: unable to parse response "{\"status\":\"success\",\"data\":{\"groups\":[{\"name\":\"CloudCredentialOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-c2ad383d-f0d6-40eb-947b-40f88973bc7a.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorDeprovisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsDeprovisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be cleaned up\",\"summary\":\"One or more CredentialsRequest CRs are unable to be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000131974,\"lastEvaluation\":\"2024-03-27T08:45:25.248120814Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorInsufficientCloudCreds\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"InsufficientCloudCreds\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.\",\"message\":\"Cluster's cloud credentials insufficient for minting or passthrough\",\"summary\":\"Problem with the available platform credentials.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000127798,\"lastEvaluation\":\"2024-03-27T08:45:25.248255358Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorProvisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsProvisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be fulfilled\",\"summary\":\"One or more CredentialsRequest CRs are unable to be processed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000126631,\"lastEvaluation\":\"2024-03-27T08:45:25.247992357Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorStaleCredentials\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"StaleCredentials\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.\",\"message\":\"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).\",\"summary\":\"One or more CredentialsRequest CRs are stale and should be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00012323,\"lastEvaluation\":\"2024-03-27T08:45:25.248384928Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorTargetNamespaceMissing\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"MissingTargetNamespace\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.\",\"message\":\"CredentialsRequest(s) pointing to non-existent namespace\",\"summary\":\"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000238978,\"lastEvaluation\":\"2024-03-27T08:45:25.247751221Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001258071,\"lastEvaluation\":\"2024-03-27T08:45:25.247200671Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-machine-approver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-fdba86e4-24de-4c05-9e26-5f58dc7abb5c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineApproverMaxPendingCSRsReached\",\"query\":\"mapi_current_pending_csr \\u003e mapi_max_pending_csr\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of pending CertificateSigningRequests has exceeded the\\nmaximum threshold (current number of machine + 100). Check the\\npending CSRs to determine which machines need approval, also check\\nthat the nodelink controller is running in the openshift-machine-api\\nnamespace.\\n\",\"summary\":\"max pending CSRs threshold reached.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00032469,\"lastEvaluation\":\"2024-03-27T08:45:36.292140978Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000339832,\"lastEvaluation\":\"2024-03-27T08:45:36.292131304Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-tuning-operator.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-6bc0735b-af85-4938-91c4-907e725cd4a8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NTODegraded\",\"query\":\"nto_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Node Tuning Operator is degraded. Review the \\\"node-tuning\\\" ClusterOperator object for further details.\",\"summary\":\"The Node Tuning Operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000204003,\"lastEvaluation\":\"2024-03-27T08:45:28.787760075Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NTOPodsNotReady\",\"query\":\"kube_pod_status_ready{condition=\\\"true\\\",namespace=\\\"openshift-cluster-node-tuning-operator\\\"} == 0\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.pod }} is not ready.\\nReview the \\\"Event\\\" objects in \\\"openshift-cluster-node-tuning-operator\\\" namespace for further details.\\n\",\"summary\":\"Pod {{ $labels.pod }} is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000599731,\"lastEvaluation\":\"2024-03-27T08:45:28.787143266Z\",\"type\":\"alerting\"},{\"name\":\"nto_custom_profiles:count\",\"query\":\"count by (_id) (nto_profile_calculated_total{profile!~\\\"openshift\\\",profile!~\\\"openshift-control-plane\\\",profile!~\\\"openshift-node\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000273454,\"lastEvaluation\":\"2024-03-27T08:45:28.788013798Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.00108666,\"lastEvaluation\":\"2024-03-27T08:45:28.787079593Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"SamplesOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7d6b51ae-913b-48fc-bbed-8261aa1794e5.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SamplesDegraded\",\"query\":\"openshift_samples_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples could not be deployed and the operator is degraded. Review the \\\"openshift-samples\\\" ClusterOperator object for further details.\\n\",\"summary\":\"Samples operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000306286,\"lastEvaluation\":\"2024-03-27T08:45:39.16708304Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesImagestreamImportFailing\",\"query\":\"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \\u003e sum(openshift_samples_failed_imagestream_import_info)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports.  You can look at the \\\"openshift-samples\\\"\\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is detecting problems with imagestream image imports\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001259634,\"lastEvaluation\":\"2024-03-27T08:45:39.165818393Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesInvalidConfig\",\"query\":\"openshift_samples_invalidconfig_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator has been given an invalid configuration.\\n\",\"summary\":\"Samples operator Invalid configuration\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00028186,\"lastEvaluation\":\"2024-03-27T08:45:39.167393914Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingSecret\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_secret\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator cannot find the samples pull secret in the openshift namespace.\\n\",\"summary\":\"Samples operator is not able to find secret\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000284289,\"lastEvaluation\":\"2024-03-27T08:45:39.167680061Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingTBRCredential\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_tbr_credential\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\\n\",\"summary\":\"Samples operator is not able to find the credentials for registry\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000265952,\"lastEvaluation\":\"2024-03-27T08:45:39.167968285Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesRetriesMissingOnImagestreamImportFailing\",\"query\":\"sum(openshift_samples_failed_imagestream_import_info) \\u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\\nimports are not occurring.  Contact support.  You can look at the \\\"openshift-samples\\\" ClusterOperator object\\nfor details. Most likely there are issues with the external image registry hosting the images that need to\\nbe investigated.  The list of ImageStreams that have failing imports are:\\n{{ range query \\\"openshift_samples_failed_imagestream_import_info \\u003e 0\\\" }}\\n  {{ .Labels.name }}\\n{{ end }}\\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is having problems with imagestream imports and its retries.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000761995,\"lastEvaluation\":\"2024-03-27T08:45:39.164894853Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesTBRInaccessibleOnBoot\",\"query\":\"openshift_samples_tbr_inaccessible_info == 1\",\"duration\":172800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"One of two situations has occurred.  Either\\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\\nassociated with each image stream tag from the samples catalog is\\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\\nassist the mirroring process.\\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\\n\",\"summary\":\"Samples operator is not able to access the registry on boot\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00030891,\"lastEvaluation\":\"2024-03-27T08:45:39.168238101Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.0018249,\"lastEvaluation\":\"2024-03-27T08:45:39.164880253Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"default-storage-classes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-2fde48fd-01e3-4ab0-8d3c-aab39c0159d4.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MultipleDefaultStorageClasses\",\"query\":\"max_over_time(default_storage_class_count[5m]) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster storage operator monitors all storage classes configured in the cluster\\nand checks there is not more than one default StorageClass configured.\\n\",\"message\":\"StorageClass count check is failing (there should not be more than one default StorageClass)\",\"summary\":\"More than one default StorageClass detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000499849,\"lastEvaluation\":\"2024-03-27T08:45:24.690196242Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000318698,\"lastEvaluation\":\"2024-03-27T08:45:24.689472838Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-2fde48fd-01e3-4ab0-8d3c-aab39c0159d4.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodStartupStorageOperationsFailing\",\"query\":\"increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status!=\\\"success\\\"}[5m]) \\u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status=\\\"success\\\"}[5m]) == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Failing storage operation \\\"{{ $labels.operation_name }}\\\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\\nfrom starting for past 5 minutes.\\nPlease investigate Pods that are \\\"ContainerCreating\\\" on the node: \\\"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\\\".\\nEvents of the Pods should contain exact error message: \\\"oc describe pod -n \\u003cpod namespace\\u003e \\u003cpod name\\u003e\\\".\\n\",\"summary\":\"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006817402,\"lastEvaluation\":\"2024-03-27T08:45:29.13735178Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003636162,\"lastEvaluation\":\"2024-03-27T08:45:29.136332985Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-operators\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-386af8f5-02f3-497a-8b16-8c3119059b4f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ClusterNotUpgradeable\",\"query\":\"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\\\"Upgradeable\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"Upgradeable\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0) (ne (len (label \\\"reason\\\" $value)) 0) }}{{label \\\"reason\\\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000491204,\"lastEvaluation\":\"2024-03-27T08:45:25.151652979Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDegraded\",\"query\":\"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\\\"Degraded\\\",job=\\\"cluster-version-operator\\\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\\\"cluster-version-operator\\\"})) == 1)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has been degraded for 30 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002183759,\"lastEvaluation\":\"2024-03-27T08:45:25.153279268Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDown\",\"query\":\"max by (namespace, name, reason) (cluster_operator_up{job=\\\"cluster-version-operator\\\"} == 0)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has not been available for 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001126798,\"lastEvaluation\":\"2024-03-27T08:45:25.152148597Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorFlapping\",\"query\":\"max by (namespace, name) (changes(cluster_operator_up{job=\\\"cluster-version-operator\\\"}[2m]) \\u003e 2)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator up status is changing often.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000871143,\"lastEvaluation\":\"2024-03-27T08:45:25.155466915Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.004698528,\"lastEvaluation\":\"2024-03-27T08:45:25.151643037Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-386af8f5-02f3-497a-8b16-8c3119059b4f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotRetrieveUpdates\",\"query\":\"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \\u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\\\"RetrievedUpdates\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\",reason!=\\\"NoChannel\\\"}))\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"RetrievedUpdates\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0)}}{{label \\\"reason\\\" $value}} {{end}}{{end}}{{end}}. {{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} For more information refer to {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/.{{ end }}{{ end }}\",\"summary\":\"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000573484,\"lastEvaluation\":\"2024-03-27T08:45:38.443436604Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterReleaseNotAccepted\",\"query\":\"max by (namespace, name, reason) (cluster_operator_conditions{condition=\\\"ReleaseAccepted\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"The desired cluster release has not been accepted for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000316833,\"lastEvaluation\":\"2024-03-27T08:45:38.444237058Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterVersionOperatorDown\",\"query\":\"absent(up{job=\\\"cluster-version-operator\\\"} == 1)\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-cluster-version\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} For more information refer to {{ label \\\"url\\\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}\",\"summary\":\"Cluster version operator has disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000356209,\"lastEvaluation\":\"2024-03-27T08:45:38.443076825Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UpdateAvailable\",\"query\":\"sum by (channel, namespace, upstream) (cluster_version_available_updates) \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Your upstream update recommendation service recommends you update your cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000218598,\"lastEvaluation\":\"2024-03-27T08:45:38.444013933Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001493469,\"lastEvaluation\":\"2024-03-27T08:45:38.443066052Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift/console-operator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-ac9f320d-b477-4dca-b2d1-9c630c99bd96.yaml\",\"rules\":[{\"name\":\"cluster:console_auth_login_failures_total:sum\",\"query\":\"sum by (reason) (console_auth_login_failures_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000182904,\"lastEvaluation\":\"2024-03-27T08:45:23.269220966Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_requests_total:sum\",\"query\":\"sum(console_auth_login_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000252171,\"lastEvaluation\":\"2024-03-27T08:45:23.268705929Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_successes_total:sum\",\"query\":\"sum by (role) (console_auth_login_successes_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0002551,\"lastEvaluation\":\"2024-03-27T08:45:23.268960858Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_logout_requests_total:sum\",\"query\":\"sum by (reason) (console_auth_logout_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000178597,\"lastEvaluation\":\"2024-03-27T08:45:23.269408522Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_customization_perspectives_info:max\",\"query\":\"max by (name, state) (console_customization_perspectives_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000129358,\"lastEvaluation\":\"2024-03-27T08:45:23.269887504Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_plugins_info:max\",\"query\":\"max by (name, state) (console_plugins_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000175232,\"lastEvaluation\":\"2024-03-27T08:45:23.269709797Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_usage_users:max\",\"query\":\"max by (role) (console_usage_users)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000117567,\"lastEvaluation\":\"2024-03-27T08:45:23.269589819Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002834942,\"lastEvaluation\":\"2024-03-27T08:45:23.25701976Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-dns.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-9c32b614-9906-4eb1-a991-5265655eef6f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CoreDNSErrorsHigh\",\"query\":\"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\\\"SERVFAIL\\\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.\",\"summary\":\"CoreDNS serverfail\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001005996,\"lastEvaluation\":\"2024-03-27T08:45:23.533650042Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSHealthCheckSlow\",\"query\":\"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \\u003e 10\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})\",\"summary\":\"CoreDNS health checks\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000861207,\"lastEvaluation\":\"2024-03-27T08:45:23.531862846Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSPanicking\",\"query\":\"increase(coredns_panics_total[10m]) \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}\",\"summary\":\"CoreDNS panic\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000390121,\"lastEvaluation\":\"2024-03-27T08:45:23.531469541Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001801079,\"lastEvaluation\":\"2024-03-27T08:45:23.531457665Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"etcd\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4b464ec4-d807-4a1f-9b84-bc7672f9c9b8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdDatabaseHighFragmentationRatio\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \\u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \\u003e 1.048576e+08\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.\",\"runbook_url\":\"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation\",\"summary\":\"etcd database size in use is less than 50% of the actual allocated storage.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000649654,\"lastEvaluation\":\"2024-03-27T08:45:46.524554715Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdDatabaseQuotaLowSpace\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \\u003e 95\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md\",\"summary\":\"etcd cluster database is running full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000488487,\"lastEvaluation\":\"2024-03-27T08:45:46.523502322Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdExcessiveDatabaseGrowth\",\"query\":\"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \\u003e etcd_server_quota_backend_bytes\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.\",\"summary\":\"etcd cluster database growing very fast.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558184,\"lastEvaluation\":\"2024-03-27T08:45:46.52399361Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighCommitDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.25\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile commit durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001379291,\"lastEvaluation\":\"2024-03-27T08:45:46.522119761Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001279039,\"lastEvaluation\":\"2024-03-27T08:45:46.520837777Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.5\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001492452,\"lastEvaluation\":\"2024-03-27T08:45:46.519342458Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedProposals\",\"query\":\"rate(etcd_server_proposals_failed_total{job=~\\\".*etcd.*\\\"}[15m]) \\u003e 5\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of proposal failures.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000386351,\"lastEvaluation\":\"2024-03-27T08:45:46.518953184Z\",\"type\":\"alerting\"},{\"state\":\"pending\",\"name\":\"etcdMemberCommunicationSlow\",\"query\":\"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.15\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster member communication is slow.\"},\"alerts\":[{\"labels\":{\"To\":\"fc0c538482edf5c3\",\"alertname\":\"etcdMemberCommunicationSlow\",\"endpoint\":\"etcd-metrics\",\"instance\":\"192.168.131.12:9979\",\"job\":\"etcd\",\"namespace\":\"openshift-etcd\",\"pod\":\"etcd-p-syd04-4xtgr-master-0\",\"service\":\"etcd\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"etcd\\\": member communication with fc0c538482edf5c3 is taking 0.1863680000000001s on etcd instance 192.168.131.12:9979.\",\"summary\":\"etcd cluster member communication is slow.\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T08:41:46.512890203Z\",\"value\":\"1.863680000000001e-01\",\"partialResponseStrategy\":\"WARN\"},{\"labels\":{\"To\":\"4e1f58f3f6f92028\",\"alertname\":\"etcdMemberCommunicationSlow\",\"endpoint\":\"etcd-metrics\",\"instance\":\"192.168.131.12:9979\",\"job\":\"etcd\",\"namespace\":\"openshift-etcd\",\"pod\":\"etcd-p-syd04-4xtgr-master-0\",\"service\":\"etcd\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"etcd\\\": member communication with 4e1f58f3f6f92028 is taking 0.1863680000000001s on etcd instance 192.168.131.12:9979.\",\"summary\":\"etcd cluster member communication is slow.\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T08:41:46.512890203Z\",\"value\":\"1.863680000000001e-01\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.003182665,\"lastEvaluation\":\"2024-03-27T08:45:46.515765726Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMembersDown\",\"query\":\"max without (endpoint) (sum without (instance) (up{job=~\\\".*etcd.*\\\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\\\".*etcd.*\\\"}[2m])) \\u003e 0.01)) \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": members are down ({{ $value }}).\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md\",\"summary\":\"etcd cluster members are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000991807,\"lastEvaluation\":\"2024-03-27T08:45:46.514499856Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdNoLeader\",\"query\":\"etcd_server_has_leader{job=~\\\".*etcd.*\\\"} == 0\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member {{ $labels.instance }} has no leader.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md\",\"summary\":\"etcd cluster has no leader.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000267827,\"lastEvaluation\":\"2024-03-27T08:45:46.515495073Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.010721875,\"lastEvaluation\":\"2024-03-27T08:45:46.514487751Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4b464ec4-d807-4a1f-9b84-bc7672f9c9b8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdGRPCRequestsSlow\",\"query\":\"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\\\"Defragment\\\",grpc_type=\\\"unary\\\",job=\\\"etcd\\\"}[10m]))) \\u003e 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md\",\"summary\":\"etcd grpc requests are slow\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.05584847,\"lastEvaluation\":\"2024-03-27T08:45:44.2668134Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 50\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.0962472,\"lastEvaluation\":\"2024-03-27T08:45:44.403877779Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 10\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.081203462,\"lastEvaluation\":\"2024-03-27T08:45:44.322669717Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfLeaderChanges\",\"query\":\"avg(changes(etcd_server_is_leader[10m])) \\u003e 5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.\",\"summary\":\"etcd cluster has high number of leader changes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000477891,\"lastEvaluation\":\"2024-03-27T08:45:44.500131619Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdInsufficientMembers\",\"query\":\"sum without (instance, pod) (up{job=\\\"etcd\\\"} == bool 1 and etcd_server_has_leader{job=\\\"etcd\\\"} == bool 1) \\u003c ((count without (instance, pod) (up{job=\\\"etcd\\\"}) + 1) / 2)\",\"duration\":180,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md\",\"summary\":\"etcd is reporting that a majority of instances are unavailable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000935162,\"lastEvaluation\":\"2024-03-27T08:45:44.500613429Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.111003446,\"lastEvaluation\":\"2024-03-27T08:45:44.26613945Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imageregistry.operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-34ed4fed-6db8-4d39-91ec-818754bb9a90.yaml\",\"rules\":[{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.Create\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000387633,\"lastEvaluation\":\"2024-03-27T08:45:26.210475488Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.ServeBlob\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00054476,\"lastEvaluation\":\"2024-03-27T08:45:26.209927518Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Get\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000360746,\"lastEvaluation\":\"2024-03-27T08:45:26.210866957Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Put\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000492941,\"lastEvaluation\":\"2024-03-27T08:45:26.211488715Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002325564,\"lastEvaluation\":\"2024-03-27T08:45:26.209661546Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imagestreams.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-3466f2a1-cd9d-4fb9-bc77-283ea39b85cb.yaml\",\"rules\":[{\"name\":\"imageregistry:imagestreamtags_count:sum\",\"query\":\"sum by (location, source) (image_registry_image_stream_tags_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000493705,\"lastEvaluation\":\"2024-03-27T08:45:40.762214503Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000514659,\"lastEvaluation\":\"2024-03-27T08:45:40.762198093Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress-to-route-controller.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-73a44dd8-1441-4959-8340-21724ebac1eb.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"IngressWithoutClassName\",\"query\":\"openshift_ingress_to_route_controller_ingress_without_class_name == 1\",\"duration\":86400,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"alerts\":[{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.129.0.56:8443\",\"job\":\"route-controller-manager\",\"name\":\"e2e-example-inghk797\",\"namespace\":\"openshift-route-controller-manager\",\"pod\":\"route-controller-manager-747675d674-79hrk\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress e2e-example-inghk797 is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T07:43:43.136557208Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"},{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.129.0.56:8443\",\"job\":\"route-controller-manager\",\"name\":\"e2e-example-ing7wmx9\",\"namespace\":\"openshift-route-controller-manager\",\"pod\":\"route-controller-manager-747675d674-79hrk\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress e2e-example-ing7wmx9 is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T07:43:43.136557208Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"},{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.129.0.56:8443\",\"job\":\"route-controller-manager\",\"name\":\"e2e-example-ingbdpl5\",\"namespace\":\"openshift-route-controller-manager\",\"pod\":\"route-controller-manager-747675d674-79hrk\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress e2e-example-ingbdpl5 is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T07:43:43.136557208Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"},{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.129.0.56:8443\",\"job\":\"route-controller-manager\",\"name\":\"test\",\"namespace\":\"openshift-route-controller-manager\",\"pod\":\"route-controller-manager-747675d674-79hrk\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress test is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T08:26:43.136557208Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000964852,\"lastEvaluation\":\"2024-03-27T08:45:43.138560461Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UnmanagedRoutes\",\"query\":\"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is a Route owned by an unmanaged Ingress.\",\"message\":\"Route {{ $labels.name }} is owned by an unmanaged Ingress.\",\"summary\":\"Route owned by an Ingress no longer managed\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002014366,\"lastEvaluation\":\"2024-03-27T08:45:43.140545507Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.004202031,\"lastEvaluation\":\"2024-03-27T08:45:43.1383644Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-73a44dd8-1441-4959-8340-21724ebac1eb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HAProxyDown\",\"query\":\"haproxy_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"This alert fires when metrics report that HAProxy is down.\",\"message\":\"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}\",\"summary\":\"HAProxy is down\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000134214,\"lastEvaluation\":\"2024-03-27T08:45:28.211383973Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HAProxyReloadFail\",\"query\":\"template_router_reload_failure == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.\",\"message\":\"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes\",\"summary\":\"HAProxy reload failure\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000254204,\"lastEvaluation\":\"2024-03-27T08:45:28.211127691Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerDegraded\",\"query\":\"ingress_controller_conditions{condition=\\\"Degraded\\\"} == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController status is degraded.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\ndegraded: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is degraded\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000150153,\"lastEvaluation\":\"2024-03-27T08:45:28.211520158Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerUnavailable\",\"query\":\"ingress_controller_conditions{condition=\\\"Available\\\"} == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController is not available.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\nunavailable: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is unavailable\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000140454,\"lastEvaluation\":\"2024-03-27T08:45:28.211672746Z\",\"type\":\"alerting\"},{\"name\":\"cluster:openshift_route_info:tls_termination:sum\",\"query\":\"sum by (tls_termination) (openshift_route_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000443213,\"lastEvaluation\":\"2024-03-27T08:45:28.212982901Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:avg\",\"query\":\"avg(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000219295,\"lastEvaluation\":\"2024-03-27T08:45:28.212446152Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:max\",\"query\":\"max(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0002276,\"lastEvaluation\":\"2024-03-27T08:45:28.212214611Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:median\",\"query\":\"quantile(0.5, route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000309002,\"lastEvaluation\":\"2024-03-27T08:45:28.212669552Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:min\",\"query\":\"min(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000341932,\"lastEvaluation\":\"2024-03-27T08:45:28.211868097Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001398715,\"lastEvaluation\":\"2024-03-27T08:45:28.211119359Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"insights\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-60fb2d31-1e44-4f26-84b3-4efa419823b6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"InsightsDisabled\",\"query\":\"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\\\"Disabled\\\",name=\\\"insights\\\"} == 1)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html\",\"summary\":\"Insights operator is disabled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000234656,\"lastEvaluation\":\"2024-03-27T08:45:23.719169132Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"InsightsRecommendationActive\",\"query\":\"insights_recommendation_active == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights recommendation \\\"{{ $labels.description }}\\\" with total risk \\\"{{ $labels.total_risk }}\\\" was detected on the cluster. More information is available at {{ $labels.info_link }}.\",\"summary\":\"An Insights recommendation is active for this cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000269124,\"lastEvaluation\":\"2024-03-27T08:45:23.71977041Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SimpleContentAccessNotAvailable\",\"query\":\"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\\\"SCAAvailable\\\",name=\\\"insights\\\",reason=\\\"NotFound\\\"}[5m]) == 0)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.\",\"summary\":\"Simple content access certificates are not available.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000228197,\"lastEvaluation\":\"2024-03-27T08:45:23.719406563Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000606696,\"lastEvaluation\":\"2024-03-27T08:45:23.719165571Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pre-release-lifecycle\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-f186c8e0-8284-4d6b-bd40-c3ac7b29650d.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"APIRemovedInNextEUSReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=~\\\"1[.]2[89]\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next EUS version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.221433684,\"lastEvaluation\":\"2024-03-27T08:45:34.754812396Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"APIRemovedInNextReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.28\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.177294582,\"lastEvaluation\":\"2024-03-27T08:45:34.577512844Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.209779822,\"lastEvaluation\":\"2024-03-27T08:45:34.577238487Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-audit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-37088249-f88e-4d34-b1f6-f14cf422071c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AuditLogError\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\\\".+-apiserver\\\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\\\".+-apiserver\\\"}[5m])) \\u003e 0\",\"duration\":60,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"An API Server had an error writing to an audit log.\",\"summary\":\"An API Server instance was unable to write audit logs. This could be\\ntriggered by the node running out of space, or a malicious actor\\ntampering with the audit logs.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000657328,\"lastEvaluation\":\"2024-03-27T08:45:31.032664265Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000671473,\"lastEvaluation\":\"2024-03-27T08:45:31.032652937Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"control-plane-cpu-utilization\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-91408eb6-f37d-442b-adbd-0aaeaaa4467f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001312669,\"lastEvaluation\":\"2024-03-27T08:45:42.787819099Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001793645,\"lastEvaluation\":\"2024-03-27T08:45:42.786022253Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneCPU\",\"query\":\"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / count(kube_node_role{role=\\\"master\\\"}) \\u003e 60\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Given three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization across all three control plane nodes is higher than two control plane nodes can sustain; a single control plane node outage may cause a cascading failure; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00154627,\"lastEvaluation\":\"2024-03-27T08:45:42.78429622Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.005290789,\"lastEvaluation\":\"2024-03-27T08:45:42.783844897Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"api-performance\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-recording-rules-719dd770-23f2-4637-8db2-a156da457e7d.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000515674,\"lastEvaluation\":\"2024-03-27T08:45:29.508686061Z\",\"type\":\"recording\"},{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000460971,\"lastEvaluation\":\"2024-03-27T08:45:29.509205218Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.086840182,\"lastEvaluation\":\"2024-03-27T08:45:28.676595832Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.097520636,\"lastEvaluation\":\"2024-03-27T08:45:28.763445182Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:1m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000460828,\"lastEvaluation\":\"2024-03-27T08:45:28.675675607Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:5m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000452751,\"lastEvaluation\":\"2024-03-27T08:45:28.676139594Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011006229,\"lastEvaluation\":\"2024-03-27T08:45:29.419004375Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011773447,\"lastEvaluation\":\"2024-03-27T08:45:29.430014807Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001664526,\"lastEvaluation\":\"2024-03-27T08:45:29.380110341Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001391514,\"lastEvaluation\":\"2024-03-27T08:45:29.383191714Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010421452,\"lastEvaluation\":\"2024-03-27T08:45:29.360215207Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009465122,\"lastEvaluation\":\"2024-03-27T08:45:29.37064083Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.019266848,\"lastEvaluation\":\"2024-03-27T08:45:29.232790735Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.021491489,\"lastEvaluation\":\"2024-03-27T08:45:29.252061712Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.042885479,\"lastEvaluation\":\"2024-03-27T08:45:29.274109399Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.043201669,\"lastEvaluation\":\"2024-03-27T08:45:29.317001778Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000304714,\"lastEvaluation\":\"2024-03-27T08:45:29.273558553Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000240156,\"lastEvaluation\":\"2024-03-27T08:45:29.273866195Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011574124,\"lastEvaluation\":\"2024-03-27T08:45:29.208941347Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.01226562,\"lastEvaluation\":\"2024-03-27T08:45:29.220520322Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012266606,\"lastEvaluation\":\"2024-03-27T08:45:29.468538518Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013907456,\"lastEvaluation\":\"2024-03-27T08:45:29.480809341Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.014539907,\"lastEvaluation\":\"2024-03-27T08:45:29.441792408Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012196271,\"lastEvaluation\":\"2024-03-27T08:45:29.45633771Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013959141,\"lastEvaluation\":\"2024-03-27T08:45:29.494722275Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000470752,\"lastEvaluation\":\"2024-03-27T08:45:28.674768603Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000430054,\"lastEvaluation\":\"2024-03-27T08:45:28.675242405Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012682469,\"lastEvaluation\":\"2024-03-27T08:45:29.058956682Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.089073325,\"lastEvaluation\":\"2024-03-27T08:45:28.860970601Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.094482662,\"lastEvaluation\":\"2024-03-27T08:45:28.950056617Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.031845253,\"lastEvaluation\":\"2024-03-27T08:45:29.072430698Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.032882006,\"lastEvaluation\":\"2024-03-27T08:45:29.104284616Z\",\"type\":\"recording\"},{\"name\":\"instance_request_kind:apiserver_current_inflight_requests:sum\",\"query\":\"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000784225,\"lastEvaluation\":\"2024-03-27T08:45:29.07164338Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.414177181,\"lastEvaluation\":\"2024-03-27T08:45:20.734265592Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.443803873,\"lastEvaluation\":\"2024-03-27T08:45:21.148452117Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":1.29954535,\"lastEvaluation\":\"2024-03-27T08:45:24.698647815Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":2.260180606,\"lastEvaluation\":\"2024-03-27T08:45:25.998202809Z\",\"type\":\"recording\"},{\"name\":\"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum\",\"query\":\"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001409904,\"lastEvaluation\":\"2024-03-27T08:45:29.381778414Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.038146048,\"lastEvaluation\":\"2024-03-27T08:45:28.527352201Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.039127423,\"lastEvaluation\":\"2024-03-27T08:45:28.565504952Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.014403873,\"lastEvaluation\":\"2024-03-27T08:45:29.044547569Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_storage_objects:max\",\"query\":\"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.017342411,\"lastEvaluation\":\"2024-03-27T08:45:29.509669434Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001288621,\"lastEvaluation\":\"2024-03-27T08:45:29.416796445Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000911925,\"lastEvaluation\":\"2024-03-27T08:45:29.418088804Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.517513932,\"lastEvaluation\":\"2024-03-27T08:45:18.90207188Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.968247259,\"lastEvaluation\":\"2024-03-27T08:45:19.766006156Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.132773105,\"lastEvaluation\":\"2024-03-27T08:45:28.258391995Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.136162991,\"lastEvaluation\":\"2024-03-27T08:45:28.391175417Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.035456861,\"lastEvaluation\":\"2024-03-27T08:45:29.137171956Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.036296998,\"lastEvaluation\":\"2024-03-27T08:45:29.172637536Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013235596,\"lastEvaluation\":\"2024-03-27T08:45:29.384586617Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.018955345,\"lastEvaluation\":\"2024-03-27T08:45:29.397833671Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":1.245159297,\"lastEvaluation\":\"2024-03-27T08:45:22.492643959Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.960829015,\"lastEvaluation\":\"2024-03-27T08:45:23.737814321Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.365888667,\"lastEvaluation\":\"2024-03-27T08:45:21.592265513Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.534468389,\"lastEvaluation\":\"2024-03-27T08:45:21.958163204Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.034557383,\"lastEvaluation\":\"2024-03-27T08:45:28.604639474Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.035558691,\"lastEvaluation\":\"2024-03-27T08:45:28.639203459Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":5.368955556,\"lastEvaluation\":\"2024-03-27T08:45:18.902060391Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-requests-in-flight\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-ba93f40a-e463-41d7-8a6a-e86ef668197d.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m\",\"query\":\"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver\\\"})[2m:])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001209815,\"lastEvaluation\":\"2024-03-27T08:45:45.551063965Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001232959,\"lastEvaluation\":\"2024-03-27T08:45:45.551047264Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-basic\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-7d793965-450f-4177-83de-ccbd862f1b2a.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1h) \\u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \\u003e (14.4 * 0.01)\",\"duration\":120,\"labels\":{\"long\":\"1h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"5m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000760572,\"lastEvaluation\":\"2024-03-27T08:45:33.637238854Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate6h) \\u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \\u003e (6 * 0.01)\",\"duration\":900,\"labels\":{\"long\":\"6h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"30m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000512383,\"lastEvaluation\":\"2024-03-27T08:45:33.638003604Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001288486,\"lastEvaluation\":\"2024-03-27T08:45:33.637231016Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-7d793965-450f-4177-83de-ccbd862f1b2a.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.280100992,\"lastEvaluation\":\"2024-03-27T08:45:26.421127785Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.500994421,\"lastEvaluation\":\"2024-03-27T08:45:27.206331523Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.148865462,\"lastEvaluation\":\"2024-03-27T08:45:26.27225396Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.343482165,\"lastEvaluation\":\"2024-03-27T08:45:27.707345404Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.104058175,\"lastEvaluation\":\"2024-03-27T08:45:26.168188559Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.104166812,\"lastEvaluation\":\"2024-03-27T08:45:28.050841587Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.505070382,\"lastEvaluation\":\"2024-03-27T08:45:26.701243396Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.223850671,\"lastEvaluation\":\"2024-03-27T08:45:28.155020473Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.512245964,\"lastEvaluation\":\"2024-03-27T08:45:30.267161329Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.551686094,\"lastEvaluation\":\"2024-03-27T08:45:29.715470426Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.539117063,\"lastEvaluation\":\"2024-03-27T08:45:29.176343408Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.391407082,\"lastEvaluation\":\"2024-03-27T08:45:28.466736194Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.318189225,\"lastEvaluation\":\"2024-03-27T08:45:28.858149387Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.046052197,\"lastEvaluation\":\"2024-03-27T08:45:28.378883423Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.041784242,\"lastEvaluation\":\"2024-03-27T08:45:28.424943577Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":4.611240772,\"lastEvaluation\":\"2024-03-27T08:45:26.168177934Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-extended\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-4d9757c2-635f-4e7b-8a9a-1d610b54092b.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1d) \\u003e (3 * 0.01) and sum(apiserver_request:burnrate2h) \\u003e (3 * 0.01)\",\"duration\":3600,\"labels\":{\"long\":\"1d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"2h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000650931,\"lastEvaluation\":\"2024-03-27T08:45:47.233474951Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate3d) \\u003e (1 * 0.01) and sum(apiserver_request:burnrate6h) \\u003e (1 * 0.01)\",\"duration\":10800,\"labels\":{\"long\":\"3d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"6h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000376846,\"lastEvaluation\":\"2024-03-27T08:45:47.234129022Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001047358,\"lastEvaluation\":\"2024-03-27T08:45:47.233462406Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-4d9757c2-635f-4e7b-8a9a-1d610b54092b.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.266518882,\"lastEvaluation\":\"2024-03-27T08:45:22.396745159Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.242420764,\"lastEvaluation\":\"2024-03-27T08:45:22.973692173Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.176789579,\"lastEvaluation\":\"2024-03-27T08:45:22.083445691Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.275907117,\"lastEvaluation\":\"2024-03-27T08:45:23.216123724Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.310400735,\"lastEvaluation\":\"2024-03-27T08:45:22.66327975Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.288269608,\"lastEvaluation\":\"2024-03-27T08:45:23.492042468Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.765764326,\"lastEvaluation\":\"2024-03-27T08:45:22.083430787Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-f87d942a-c815-4a99-b93b-7f0a0b4292bb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TechPreviewNoUpgrade\",\"query\":\"cluster_feature_set{name!=\\\"\\\",namespace=\\\"openshift-kube-apiserver-operator\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.\",\"summary\":\"Cluster has enabled tech preview features that will prevent upgrades.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006021682,\"lastEvaluation\":\"2024-03-27T08:45:25.091445268Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002873574,\"lastEvaluation\":\"2024-03-27T08:45:25.091178929Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pod-security-violation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-f71e3e7e-2d2a-4ae6-96a7-b23c385f9bc8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace!=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created in namespace \\\"{{ $labels.ocp_namespace }}\\\" but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00025296,\"lastEvaluation\":\"2024-03-27T08:45:31.457305558Z\",\"type\":\"alerting\"},{\"state\":\"firing\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads users created in the cluster don't match their Pod Security profile\"},\"alerts\":[{\"labels\":{\"alertname\":\"PodSecurityViolation\",\"namespace\":\"openshift-kube-apiserver\",\"policy_level\":\"restricted\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \\\"restricted\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads users created in the cluster don't match their Pod Security profile\"},\"state\":\"firing\",\"activeAt\":\"2024-03-27T08:26:31.452115983Z\",\"value\":\"1.0347239285714286e+01\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.001061565,\"lastEvaluation\":\"2024-03-27T08:45:31.45624014Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001335471,\"lastEvaluation\":\"2024-03-27T08:45:31.456227129Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-6a4bea6d-69ac-4b07-8ab7-34e717b065d0.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"GarbageCollectorSyncFailed\",\"query\":\"rate(garbagecollector_controller_resources_sync_error_total[5m]) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md\",\"summary\":\"There was a problem with syncing the resources for garbage collection.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000481275,\"lastEvaluation\":\"2024-03-27T08:45:20.804146002Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeControllerManagerDown\",\"query\":\"absent(up{job=\\\"kube-controller-manager\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-controller-manager\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeControllerManager has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000505358,\"lastEvaluation\":\"2024-03-27T08:45:20.799902613Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetAtLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \\u003e 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md\",\"summary\":\"The pod disruption budget is preventing further disruption to pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001685176,\"lastEvaluation\":\"2024-03-27T08:45:20.800415406Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \\u003c kube_poddisruptionbudget_status_desired_healthy)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md\",\"summary\":\"The pod disruption budget registers insufficient amount of pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00203341,\"lastEvaluation\":\"2024-03-27T08:45:20.802106303Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002049633,\"lastEvaluation\":\"2024-03-27T08:45:20.799772292Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-cd5bc467-4fd9-4006-9cdb-67ca18b3a004.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeSchedulerDown\",\"query\":\"absent(up{job=\\\"scheduler\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-scheduler\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeScheduler has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000215893,\"lastEvaluation\":\"2024-03-27T08:45:43.160710229Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000463713,\"lastEvaluation\":\"2024-03-27T08:45:43.160009979Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"scheduler-legacy-policy-deprecated\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-cd5bc467-4fd9-4006-9cdb-67ca18b3a004.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SchedulerLegacyPolicySet\",\"query\":\"cluster_legacy_scheduler_policy \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.\",\"summary\":\"Legacy scheduler policy API in use by the scheduler.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000243046,\"lastEvaluation\":\"2024-03-27T08:45:32.913103609Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000657758,\"lastEvaluation\":\"2024-03-27T08:45:32.912920123Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-api-operator-metrics-collector-up\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineAPIOperatorMetricsCollectionFailing\",\"query\":\"mapi_mao_collector_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"For more details:  oc logs \\u003cmachine-api-operator-pod-name\\u003e -n openshift-machine-api\",\"summary\":\"machine api operator metrics collection is failing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000859831,\"lastEvaluation\":\"2024-03-27T08:45:27.3535061Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000364509,\"lastEvaluation\":\"2024-03-27T08:45:27.347706727Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-health-check-unterminated-short-circuit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineHealthCheckUnterminatedShortCircuit\",\"query\":\"mapi_machinehealthcheck_short_circuit == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\\nthe status of machines in the cluster.\\n\",\"summary\":\"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000362663,\"lastEvaluation\":\"2024-03-27T08:45:32.708101452Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000406098,\"lastEvaluation\":\"2024-03-27T08:45:32.708070201Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-not-yet-deleted\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineNotYetDeleted\",\"query\":\"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\\\"Deleting\\\"}[15m])) \\u003e 0\",\"duration\":21600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine is not properly deleting, this may be due to a configuration issue with the\\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\\nlong termination periods which are preventing deletion.\\n\",\"summary\":\"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000347677,\"lastEvaluation\":\"2024-03-27T08:45:39.596140743Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000564677,\"lastEvaluation\":\"2024-03-27T08:45:39.596103747Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-with-no-running-phase\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithNoRunningPhase\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\\\"Running|Deleting\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine has been without a Running or Deleting phase for more than 60 minutes.\\nThe machine may not have been provisioned properly from the infrastructure provider, or\\nit might have issues with CertificateSigningRequests being approved.\\n\",\"summary\":\"machine {{ $labels.name }} is in phase: {{ $labels.phase }}\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000222833,\"lastEvaluation\":\"2024-03-27T08:45:24.892835516Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000785693,\"lastEvaluation\":\"2024-03-27T08:45:24.892718281Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-without-valid-node-ref\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithoutValidNode\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"If the machine never became a node, you should diagnose the machine related failures.\\nIf the node was deleted from the API, you may delete the machine if appropriate.\\n\",\"summary\":\"machine {{ $labels.name }} does not have valid node reference\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000791042,\"lastEvaluation\":\"2024-03-27T08:45:28.445891363Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000810741,\"lastEvaluation\":\"2024-03-27T08:45:28.445880983Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-drain-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-c3ddad37-88e0-4b4a-a21a-ef5f971392f8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCDrainError\",\"query\":\"mcc_drain_err \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000324526,\"lastEvaluation\":\"2024-03-27T08:45:36.421585265Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000231023,\"lastEvaluation\":\"2024-03-27T08:45:36.421472523Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-pool-alert\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-c3ddad37-88e0-4b4a-a21a-ef5f971392f8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCPoolAlert\",\"query\":\"mcc_pool_alert \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000577471,\"lastEvaluation\":\"2024-03-27T08:45:24.443110855Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000545323,\"lastEvaluation\":\"2024-03-27T08:45:24.44264836Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"os-image-override.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-c3ddad37-88e0-4b4a-a21a-ef5f971392f8.yaml\",\"rules\":[{\"name\":\"os_image_url_override:sum\",\"query\":\"sum(os_image_url_override)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000607114,\"lastEvaluation\":\"2024-03-27T08:45:42.659325222Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000431801,\"lastEvaluation\":\"2024-03-27T08:45:42.658780681Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"extremely-high-individual-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneMemory\",\"query\":\"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 90\",\"duration\":2700,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.\",\"summary\":\"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00222741,\"lastEvaluation\":\"2024-03-27T08:45:26.130717627Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001381396,\"lastEvaluation\":\"2024-03-27T08:45:26.129816223Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"high-overall-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneMemory\",\"query\":\"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 60\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.\",\"summary\":\"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00108057,\"lastEvaluation\":\"2024-03-27T08:45:39.842642699Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001095726,\"lastEvaluation\":\"2024-03-27T08:45:39.842630014Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-kubelet-health-state-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeletHealthState\",\"query\":\"mcd_kubelet_state \\u003e 2\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet health failure threshold reached\",\"summary\":\"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000606619,\"lastEvaluation\":\"2024-03-27T08:45:33.974737203Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000629891,\"lastEvaluation\":\"2024-03-27T08:45:33.974719954Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-pivot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDPivotError\",\"query\":\"mcd_pivot_errors_total \\u003e 0\",\"duration\":120,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000276504,\"lastEvaluation\":\"2024-03-27T08:45:31.24104656Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000286744,\"lastEvaluation\":\"2024-03-27T08:45:31.241039088Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-reboot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDRebootError\",\"query\":\"mcd_reboots_failed_total \\u003e 0\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000289734,\"lastEvaluation\":\"2024-03-27T08:45:44.852829304Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000302216,\"lastEvaluation\":\"2024-03-27T08:45:44.852819749Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"system-memory-exceeds-reservation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SystemMemoryExceedsReservation\",\"query\":\"sum by (node) (container_memory_rss{id=\\\"/system.slice\\\"}) \\u003e ((sum by (node) (kube_node_status_capacity{resource=\\\"memory\\\"} - kube_node_status_allocatable{resource=\\\"memory\\\"})) * 0.95)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).\",\"summary\":\"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000941558,\"lastEvaluation\":\"2024-03-27T08:45:31.527339494Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00095989,\"lastEvaluation\":\"2024-03-27T08:45:31.527325789Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"operator.marketplace.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-marketplace-marketplace-alert-rules-5f4ea3a5-aee7-484b-96bc-522d41b231cc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"OperatorHubSourceError\",\"query\":\"catalogsource_ready{exported_namespace=\\\"openshift-marketplace\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Operators shipped via the {{ $labels.name }} source are not available for installation until the issue is fixed. Operators already installed from this source will not receive updates until issue is fixed. Inspect the status of the pod owned by {{ $labels.name }} source in the openshift-marketplace namespace (oc -n openshift-marketplace get pods -l olm.catalogSource={{ $labels.name }}) to diagnose and repair.\",\"summary\":\"The {{ $labels.name }} source is in non-ready state for more than 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000743302,\"lastEvaluation\":\"2024-03-27T08:45:42.595064825Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000405628,\"lastEvaluation\":\"2024-03-27T08:45:42.594435753Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"alertmanager.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-alertmanager-main-rules-0472e654-98ee-4d67-8369-8aebd739a6d2.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterDown\",\"query\":\"(count by (namespace, service) (avg_over_time(up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c 0.5) / count by (namespace, service) (up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) \\u003e= 0.5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of Alertmanager instances within the {{$labels.job}} cluster have been up for less than half of the last 5m.\",\"summary\":\"Half or more of the Alertmanager instances within the same cluster are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000890144,\"lastEvaluation\":\"2024-03-27T08:45:29.251030565Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterFailedToSendAlerts\",\"query\":\"min by (namespace, service, integration) (rate(alertmanager_notifications_failed_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / rate(alertmanager_notifications_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The minimum notification failure rate to {{ $labels.integration }} sent from any instance in the {{$labels.job}} cluster is {{ $value | humanizePercentage }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerClusterFailedToSendAlerts.md\",\"summary\":\"All Alertmanager instances in a cluster failed to send notifications to a critical integration.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001968288,\"lastEvaluation\":\"2024-03-27T08:45:29.248564779Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerConfigInconsistent\",\"query\":\"count by (namespace, service) (count_values by (namespace, service) (\\\"config_hash\\\", alertmanager_config_hash{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) != 1\",\"duration\":1200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager instances within the {{$labels.job}} cluster have different configurations.\",\"summary\":\"Alertmanager instances within the same cluster have different configurations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000489018,\"lastEvaluation\":\"2024-03-27T08:45:29.250537514Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedReload\",\"query\":\"max_over_time(alertmanager_config_last_reload_successful{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Configuration has failed to load for {{ $labels.namespace }}/{{ $labels.pod}}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedReload.md\",\"summary\":\"Reloading an Alertmanager configuration has failed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000344675,\"lastEvaluation\":\"2024-03-27T08:45:29.245727162Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedToSendAlerts\",\"query\":\"(rate(alertmanager_notifications_failed_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / rate(alertmanager_notifications_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} failed to send {{ $value | humanizePercentage }} of notifications to {{ $labels.integration }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedToSendAlerts.md\",\"summary\":\"An Alertmanager instance failed to send notifications.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001872041,\"lastEvaluation\":\"2024-03-27T08:45:29.246688486Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerMembersInconsistent\",\"query\":\"max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c on (namespace, service) group_left () count by (namespace, service) (max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]))\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} has only found {{ $value }} members of the {{$labels.job}} cluster.\",\"summary\":\"A member of an Alertmanager cluster has not found all other cluster members.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000347718,\"lastEvaluation\":\"2024-03-27T08:45:29.246074025Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002991198,\"lastEvaluation\":\"2024-03-27T08:45:29.245718092Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-list-watch.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"apiserver_list_watch_request_success_total:rate:sum\",\"query\":\"sum by (verb) (rate(apiserver_request_total{code=~\\\"2..\\\",verb=~\\\"LIST|WATCH\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.026909,\"lastEvaluation\":\"2024-03-27T08:45:45.807634848Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.04322445,\"lastEvaluation\":\"2024-03-27T08:45:45.806812425Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"Watchdog\",\"query\":\"vector(1)\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"alerts\":[{\"labels\":{\"alertname\":\"Watchdog\",\"namespace\":\"openshift-monitoring\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"state\":\"firing\",\"activeAt\":\"2024-03-27T07:22:21.164916952Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000690437,\"lastEvaluation\":\"2024-03-27T08:45:21.166845634Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000709195,\"lastEvaluation\":\"2024-03-27T08:45:21.166833447Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"count:up0\",\"query\":\"count without (instance, pod, node) (up == 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002950463,\"lastEvaluation\":\"2024-03-27T08:45:32.981802677Z\",\"type\":\"recording\"},{\"name\":\"count:up1\",\"query\":\"count without (instance, pod, node) (up == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003070788,\"lastEvaluation\":\"2024-03-27T08:45:32.978286212Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006569208,\"lastEvaluation\":\"2024-03-27T08:45:32.97818814Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-node-recording.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"cluster:node_cpu:ratio\",\"query\":\"cluster:node_cpu:sum_rate5m / count(sum by (instance, cpu) (node_cpu_seconds_total))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009158959,\"lastEvaluation\":\"2024-03-27T08:45:28.874578197Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:sum_rate5m\",\"query\":\"sum(rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006759567,\"lastEvaluation\":\"2024-03-27T08:45:28.867811301Z\",\"type\":\"recording\"},{\"name\":\"instance:node_cpu:rate:sum\",\"query\":\"sum by (instance) (rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002691087,\"lastEvaluation\":\"2024-03-27T08:45:28.859294055Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_receive_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001294304,\"lastEvaluation\":\"2024-03-27T08:45:28.865235684Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_transmit_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_transmit_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001272228,\"lastEvaluation\":\"2024-03-27T08:45:28.866534869Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.025226908,\"lastEvaluation\":\"2024-03-27T08:45:28.858518251Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-recurring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"cluster:usage:workload:capacity_physical_cpu_core_seconds\",\"query\":\"sum_over_time(workload:capacity_physical_cpu_cores:sum[30s:1s]) + ((cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) or (absent(cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) * 0))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001044034,\"lastEvaluation\":\"2024-03-27T08:45:41.448431901Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001067658,\"lastEvaluation\":\"2024-03-27T08:45:41.448415923Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-network\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeNetworkInterfaceFlapping\",\"query\":\"changes(node_network_up{device!~\\\"veth.+|tunbr\\\",job=\\\"node-exporter\\\"}[2m]) \\u003e 2\",\"duration\":120,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Network interface \\\"{{ $labels.device }}\\\" changing its up status often on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\",\"summary\":\"Network interface is often changing its status\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001129163,\"lastEvaluation\":\"2024-03-27T08:45:30.225782206Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001138163,\"lastEvaluation\":\"2024-03-27T08:45:30.225777416Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-build.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"openshift:build_by_strategy:sum\",\"query\":\"sum by (strategy) (openshift_build_status_phase_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000463385,\"lastEvaluation\":\"2024-03-27T08:45:19.288247594Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000579121,\"lastEvaluation\":\"2024-03-27T08:45:19.287958958Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd-telemetry.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001371138,\"lastEvaluation\":\"2024-03-27T08:45:31.534270809Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001900661,\"lastEvaluation\":\"2024-03-27T08:45:31.529373116Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000384632,\"lastEvaluation\":\"2024-03-27T08:45:31.528983793Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_use_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00026768,\"lastEvaluation\":\"2024-03-27T08:45:31.533999966Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.002717597,\"lastEvaluation\":\"2024-03-27T08:45:31.531278925Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006666372,\"lastEvaluation\":\"2024-03-27T08:45:31.528979025Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TargetDown\",\"query\":\"100 * ((1 - sum by (job, namespace, service) (up and on (namespace, pod) kube_pod_info) / count by (job, namespace, service) (up and on (namespace, pod) kube_pod_info)) or (count by (job, namespace, service) (up == 0) / count by (job, namespace, service) (up))) \\u003e 10\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ printf \\\"%.4g\\\" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.\",\"summary\":\"Some targets were not reachable from the monitoring server for an extended period of time.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.034926089,\"lastEvaluation\":\"2024-03-27T08:45:38.637381409Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.020686266,\"lastEvaluation\":\"2024-03-27T08:45:38.637345854Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"cluster:ingress_controller_aws_nlb_active:sum\",\"query\":\"sum(ingress_controller_aws_nlb_active) or vector(0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000607386,\"lastEvaluation\":\"2024-03-27T08:45:24.898300637Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_in:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_in_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000885292,\"lastEvaluation\":\"2024-03-27T08:45:24.859711714Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_out:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_out_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000832808,\"lastEvaluation\":\"2024-03-27T08:45:24.860602908Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_connections:sum\",\"query\":\"sum(haproxy_frontend_current_sessions)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000622141,\"lastEvaluation\":\"2024-03-27T08:45:24.861444328Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace=~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.015124351,\"lastEvaluation\":\"2024-03-27T08:45:24.870711839Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.01244188,\"lastEvaluation\":\"2024-03-27T08:45:24.885846178Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace!~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005143992,\"lastEvaluation\":\"2024-03-27T08:45:24.862073765Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003480493,\"lastEvaluation\":\"2024-03-27T08:45:24.867223962Z\",\"type\":\"recording\"},{\"name\":\"code:cluster:ingress_http_request_count:rate5m:sum\",\"query\":\"sum by (code) (rate(haproxy_server_http_responses_total[5m]) \\u003e 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010005955,\"lastEvaluation\":\"2024-03-27T08:45:24.849696854Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.049231665,\"lastEvaluation\":\"2024-03-27T08:45:24.849683427Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-kubernetes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"AlertmanagerReceiversNotConfigured\",\"query\":\"cluster:alertmanager_integrations:max == 0\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"alerts\":[{\"labels\":{\"alertname\":\"AlertmanagerReceiversNotConfigured\",\"namespace\":\"openshift-monitoring\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"state\":\"firing\",\"activeAt\":\"2024-03-27T07:22:38.472010051Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000469268,\"lastEvaluation\":\"2024-03-27T08:45:38.909460701Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterMonitoringOperatorReconciliationErrors\",\"query\":\"max_over_time(cluster_monitoring_operator_last_reconciliation_successful[5m]) == 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors are occurring during reconciliation cycles. Inspect the cluster-monitoring-operator log for potential root causes.\",\"summary\":\"Cluster Monitoring Operator is experiencing unexpected reconciliation errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000223699,\"lastEvaluation\":\"2024-03-27T08:45:38.909234087Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentReplicasMismatch\",\"query\":\"(((kube_deployment_spec_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_deployment_status_replicas_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_deployment_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)) * on () group_left () cluster:control_plane:all_nodes_ready) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes. This indicates that cluster infrastructure is unable to start or restart the necessary components. This most often occurs when one or more nodes are down or partioned from the cluster, or a fault occurs on the node that prevents the workload from starting. In rare cases this may indicate a new version of a cluster component cannot start due to a bug or configuration error. Assess the pods for this deployment to verify they are running on healthy nodes and then contact support.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeDeploymentReplicasMismatch.md\",\"summary\":\"Deployment has not matched the expected number of replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00864537,\"lastEvaluation\":\"2024-03-27T08:45:38.909933175Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodNotScheduled\",\"query\":\"last_over_time(kube_pod_status_unschedulable{namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} cannot be scheduled for more than 30 minutes.\\nCheck the details of the pod with the following command:\\noc describe -n {{ $labels.namespace }} pod {{ $labels.pod }}\",\"summary\":\"Pod cannot be scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00122576,\"lastEvaluation\":\"2024-03-27T08:45:39.018697044Z\",\"type\":\"alerting\"},{\"name\":\"cluster:alertmanager_integrations:max\",\"query\":\"max(alertmanager_integrations{namespace=\\\"openshift-monitoring\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00020203,\"lastEvaluation\":\"2024-03-27T08:45:38.907969987Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (label_replace(cluster:infra_nodes, \\\"label_node_role_kubernetes_io\\\", \\\"infra\\\", \\\"\\\", \\\"\\\") * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001324817,\"lastEvaluation\":\"2024-03-27T08:45:38.77677198Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000787115,\"lastEvaluation\":\"2024-03-27T08:45:38.783383305Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_sockets_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (max by (node, package, label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00089522,\"lastEvaluation\":\"2024-03-27T08:45:38.906161087Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_memory_bytes:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000918405,\"lastEvaluation\":\"2024-03-27T08:45:38.784173747Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_cpu_usage:ratio\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m])) / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.042044936,\"lastEvaluation\":\"2024-03-27T08:45:38.730962937Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_spec_cpu_shares:ratio\",\"query\":\"sum(container_spec_cpu_shares{container=\\\"\\\",pod!=\\\"\\\"}) / 1000 / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.044153161,\"lastEvaluation\":\"2024-03-27T08:45:38.686805032Z\",\"type\":\"recording\"},{\"name\":\"cluster:control_plane:all_nodes_ready\",\"query\":\"sum(min by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}) and max by (node) (kube_node_role{role=\\\"master\\\"})) == bool sum(kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000558666,\"lastEvaluation\":\"2024-03-27T08:45:38.908482872Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_hyperthreading\",\"query\":\"clamp_max(label_replace(sum by (instance, package, core) (node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_replace(label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003e 1, \\\"label_node_hyperthread_enabled\\\", \\\"true\\\", \\\"instance\\\", \\\"(.*)\\\") or on (instance, package) label_replace(sum by (instance, package, core) (label_replace(node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003c= 1, \\\"label_node_hyperthread_enabled\\\", \\\"false\\\", \\\"instance\\\", \\\"(.*)\\\"), 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003599898,\"lastEvaluation\":\"2024-03-27T08:45:38.778100353Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_node_labels\",\"query\":\"topk by (node) (1, cluster:nodes_roles) * on (node) group_right (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_node_openshift_io_os_id, label_kubernetes_io_arch, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) label_replace(cluster:cpu_core_hyperthreading, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00167668,\"lastEvaluation\":\"2024-03-27T08:45:38.781703512Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_usage_cores:sum\",\"query\":\"sum(1 - rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[2m]) * on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{pod=~\\\"node-exporter.+\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001906469,\"lastEvaluation\":\"2024-03-27T08:45:38.785095177Z\",\"type\":\"recording\"},{\"name\":\"cluster:hyperthread_enabled_nodes\",\"query\":\"kube_node_labels and on (node) (sum by (node, package, core) (label_replace(node_cpu_info, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")) == 2)\",\"labels\":{\"label_node_hyperthread_enabled\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001765164,\"lastEvaluation\":\"2024-03-27T08:45:38.774603832Z\",\"type\":\"recording\"},{\"name\":\"cluster:infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"infra\\\"})\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00030626,\"lastEvaluation\":\"2024-03-27T08:45:38.773462579Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolume_plugin_type_counts:sum\",\"query\":\"sum by (plugin_name, volume_mode) (pv_collector_total_pv_count)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000304869,\"lastEvaluation\":\"2024-03-27T08:45:38.90817509Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_resource_requests_storage_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001485419,\"lastEvaluation\":\"2024-03-27T08:45:38.875286571Z\",\"type\":\"recording\"},{\"name\":\"cluster:kubelet_volume_stats_used_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kubelet_volume_stats_used_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001144583,\"lastEvaluation\":\"2024-03-27T08:45:38.878217459Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (cluster:master_nodes and on (node) cluster:infra_nodes)\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000265586,\"lastEvaluation\":\"2024-03-27T08:45:38.773771756Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"label_node_role_kubernetes_io\":\"master\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000446442,\"lastEvaluation\":\"2024-03-27T08:45:38.773012938Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage:ratio\",\"query\":\"sum by (cluster) (container_memory_usage_bytes{container=\\\"\\\",pod!=\\\"\\\"}) / sum by (cluster) (machine_memory_bytes)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.046905606,\"lastEvaluation\":\"2024-03-27T08:45:38.63989404Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage_bytes:sum\",\"query\":\"sum(node_memory_MemTotal_bytes{job=\\\"node-exporter\\\"} - node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000452129,\"lastEvaluation\":\"2024-03-27T08:45:38.787004775Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_instance_type_count:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) (cluster:master_nodes or on (node) kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000404111,\"lastEvaluation\":\"2024-03-27T08:45:38.874879372Z\",\"type\":\"recording\"},{\"name\":\"cluster:nodes_roles\",\"query\":\"cluster:master_infra_nodes or on (node) cluster:master_nodes or on (node) cluster:infra_nodes or on (node) max without (endpoint, instance, job, pod, service) (kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000560411,\"lastEvaluation\":\"2024-03-27T08:45:38.774040114Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:containers:sum\",\"query\":\"sum(max by (instance) (kubelet_containers_per_pod_count_sum))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00045936,\"lastEvaluation\":\"2024-03-27T08:45:38.904542567Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_node_ready:avg5m\",\"query\":\"avg_over_time((count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1)) / scalar(count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010728877,\"lastEvaluation\":\"2024-03-27T08:45:38.946202011Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_schedulable_node_ready_reachable:avg5m\",\"query\":\"avg_over_time((((count((max by (node) (up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1) and max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1) and min by (node) (kube_node_spec_unschedulable == 0))) / scalar(count(min by (node) (kube_node_spec_unschedulable == 0))))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.027611442,\"lastEvaluation\":\"2024-03-27T08:45:38.918582096Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace=~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005208091,\"lastEvaluation\":\"2024-03-27T08:45:39.008385243Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:pods:terminal:workload:sum\",\"query\":\"count(count by (namespace, pod) (kube_pod_restart_policy{namespace!~\\\"openshift-.+\\\",type!=\\\"Always\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000768445,\"lastEvaluation\":\"2024-03-27T08:45:38.903768999Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:resources:sum\",\"query\":\"topk(500, max by (resource) (apiserver_storage_objects))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012878969,\"lastEvaluation\":\"2024-03-27T08:45:38.890884116Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:max:5m\",\"query\":\"max_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000190541,\"lastEvaluation\":\"2024-03-27T08:45:38.878023737Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:min:5m\",\"query\":\"min_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000211438,\"lastEvaluation\":\"2024-03-27T08:45:38.877809234Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace!~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005095546,\"lastEvaluation\":\"2024-03-27T08:45:39.013597457Z\",\"type\":\"recording\"},{\"name\":\"cluster:virt_platform_nodes:sum\",\"query\":\"count by (type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (sum by (instance, type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (virt_platform))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000394327,\"lastEvaluation\":\"2024-03-27T08:45:38.776374274Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_object_counts:sum\",\"query\":\"sum by (instance) (apiserver_storage_objects)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011515503,\"lastEvaluation\":\"2024-03-27T08:45:38.879365075Z\",\"type\":\"recording\"},{\"name\":\"kube_running_pod_ready\",\"query\":\"(max without (condition, container, endpoint, instance, job, service) (((kube_pod_status_ready{condition=\\\"false\\\"} == 1) * 0 or (kube_pod_status_ready{condition=\\\"true\\\"} == 1)) * on (pod, namespace) group_left () group by (pod, namespace) (kube_pod_status_phase{phase=~\\\"Running|Unknown|Pending\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.051443961,\"lastEvaluation\":\"2024-03-27T08:45:38.956935245Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_cpu_usage:sum\",\"query\":\"sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=\\\"\\\",container!=\\\"POD\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.026301636,\"lastEvaluation\":\"2024-03-27T08:45:38.613586089Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_memory_usage_bytes:sum\",\"query\":\"sum by (namespace) (container_memory_usage_bytes{container!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.043659328,\"lastEvaluation\":\"2024-03-27T08:45:38.569918488Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_cores:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001150362,\"lastEvaluation\":\"2024-03-27T08:45:38.905006594Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_sockets:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (max by (node, package, label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000907233,\"lastEvaluation\":\"2024-03-27T08:45:38.907059562Z\",\"type\":\"recording\"},{\"name\":\"openshift:cpu_usage_cores:sum\",\"query\":\"cluster:cpu_usage_cores:sum - workload:cpu_usage_cores:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000341643,\"lastEvaluation\":\"2024-03-27T08:45:38.829965851Z\",\"type\":\"recording\"},{\"name\":\"openshift:memory_usage_bytes:sum\",\"query\":\"cluster:memory_usage_bytes:sum - workload:memory_usage_bytes:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000321816,\"lastEvaluation\":\"2024-03-27T08:45:38.87455433Z\",\"type\":\"recording\"},{\"name\":\"pod:container_cpu_usage:sum\",\"query\":\"sum by (pod, namespace) (rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.027299801,\"lastEvaluation\":\"2024-03-27T08:45:38.473614406Z\",\"type\":\"recording\"},{\"name\":\"pod:container_fs_usage_bytes:sum\",\"query\":\"sum by (pod, namespace) (container_fs_usage_bytes{pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.05082194,\"lastEvaluation\":\"2024-03-27T08:45:38.519080827Z\",\"type\":\"recording\"},{\"name\":\"profile:cluster_monitoring_operator_collection_profile:max\",\"query\":\"max by (profile) (cluster_monitoring_operator_collection_profile == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000185952,\"lastEvaluation\":\"2024-03-27T08:45:38.909044859Z\",\"type\":\"recording\"},{\"name\":\"workload:capacity_physical_cpu_cores:sum\",\"query\":\"(sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_infra=\\\"\\\",label_node_role_kubernetes_io_master=\\\"\\\"} or absent(__does_not_exist__) * 0)) + ((sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_master=\\\"true\\\"} or absent(__does_not_exist__) * 0) * ((max(cluster_master_schedulable == 1) * 0 + 1) or (absent(cluster_master_schedulable == 1) * 0))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001030688,\"lastEvaluation\":\"2024-03-27T08:45:38.876775288Z\",\"type\":\"recording\"},{\"name\":\"workload:cpu_usage_cores:sum\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.042499417,\"lastEvaluation\":\"2024-03-27T08:45:38.787460008Z\",\"type\":\"recording\"},{\"name\":\"workload:memory_usage_bytes:sum\",\"query\":\"sum(container_memory_working_set_bytes{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.044238413,\"lastEvaluation\":\"2024-03-27T08:45:38.830310619Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.316973138,\"lastEvaluation\":\"2024-03-27T08:45:38.473601633Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-monitoring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"monitoring:container_memory_working_set_bytes:sum\",\"query\":\"sum by (namespace) (max without (instance) (container_memory_working_set_bytes{container=\\\"\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003619478,\"lastEvaluation\":\"2024-03-27T08:45:31.779843038Z\",\"type\":\"recording\"},{\"name\":\"monitoring:haproxy_server_http_responses_total:sum\",\"query\":\"sum by (exported_service) (rate(haproxy_server_http_responses_total{exported_namespace=\\\"openshift-monitoring\\\",exported_service=~\\\"alertmanager-main|prometheus-k8s\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002337153,\"lastEvaluation\":\"2024-03-27T08:45:31.799866279Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_samples_post_metric_relabeling:topk3\",\"query\":\"topk(3, max by (namespace, job) (topk by (namespace, job) (1, scrape_samples_post_metric_relabeling)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003999327,\"lastEvaluation\":\"2024-03-27T08:45:31.795863284Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_series_added:topk3_sum1h\",\"query\":\"topk(3, sum by (namespace, job) (sum_over_time(scrape_series_added[1h])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012390659,\"lastEvaluation\":\"2024-03-27T08:45:31.783467336Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicationController\\\"}, \\\"replicationcontroller\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicationcontroller, namespace) group_left (owner_name) topk by (replicationcontroller, namespace) (1, max by (replicationcontroller, namespace, owner_name) (kube_replicationcontroller_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deploymentconfig\"},\"health\":\"ok\",\"evaluationTime\":0.002530907,\"lastEvaluation\":\"2024-03-27T08:45:31.802208423Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_samples_appended_total:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (rate(prometheus_tsdb_head_samples_appended_total{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[2m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000398768,\"lastEvaluation\":\"2024-03-27T08:45:31.779437589Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_series:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (prometheus_tsdb_head_series{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000521319,\"lastEvaluation\":\"2024-03-27T08:45:31.778912411Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.012781179,\"lastEvaluation\":\"2024-03-27T08:45:31.778506926Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-sre.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"code:apiserver_request_total:rate:sum\",\"query\":\"sum by (code) (rate(apiserver_request_total{job=\\\"apiserver\\\"}[10m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.086422803,\"lastEvaluation\":\"2024-03-27T08:45:30.036743364Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.049049596,\"lastEvaluation\":\"2024-03-27T08:45:30.036704345Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-vsphere.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"cluster:vsphere_csi_migration:max\",\"query\":\"max by (status) (vsphere_csi_migration{status=~\\\"|LegacyDeprecatedInTreeDriver|CSIWithMigrationDriver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000568996,\"lastEvaluation\":\"2024-03-27T08:45:22.403011826Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_esxi_version_total:sum\",\"query\":\"sum by (version) (vsphere_esxi_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000120959,\"lastEvaluation\":\"2024-03-27T08:45:22.402385945Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_infrastructure_failure_domains:max\",\"query\":\"max by (scope) (vsphere_infrastructure_failure_domains)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000119476,\"lastEvaluation\":\"2024-03-27T08:45:22.402768256Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_node_hw_version_total:sum\",\"query\":\"sum by (hw_version) (vsphere_node_hw_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000128957,\"lastEvaluation\":\"2024-03-27T08:45:22.402510177Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_topology_tags:max\",\"query\":\"max by (source) (vsphere_topology_tags)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000123939,\"lastEvaluation\":\"2024-03-27T08:45:22.402641653Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_vcenter_info:sum\",\"query\":\"sum by (version, build) (vsphere_vcenter_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000246378,\"lastEvaluation\":\"2024-03-27T08:45:22.402136945Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000973216,\"lastEvaluation\":\"2024-03-27T08:45:22.402133125Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-state-metrics\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kube-state-metrics-rules-efd623b1-12df-4ae4-ba23-205c862f8327.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeStateMetricsListErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in list operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in list operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000664315,\"lastEvaluation\":\"2024-03-27T08:45:33.620853113Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStateMetricsWatchErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in watch operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in watch operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000513111,\"lastEvaluation\":\"2024-03-27T08:45:33.621522092Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00251924,\"lastEvaluation\":\"2024-03-27T08:45:33.619519977Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.038487375,\"lastEvaluation\":\"2024-03-27T08:45:23.70772229Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.055410374,\"lastEvaluation\":\"2024-03-27T08:45:23.514685Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.037652177,\"lastEvaluation\":\"2024-03-27T08:45:23.635339597Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.056286061,\"lastEvaluation\":\"2024-03-27T08:45:23.40344913Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.035085797,\"lastEvaluation\":\"2024-03-27T08:45:23.74621606Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.065220889,\"lastEvaluation\":\"2024-03-27T08:45:23.570106597Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.03471478,\"lastEvaluation\":\"2024-03-27T08:45:23.67299929Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.054930196,\"lastEvaluation\":\"2024-03-27T08:45:23.459745233Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"DaemonSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"daemonset\"},\"health\":\"ok\",\"evaluationTime\":0.002960942,\"lastEvaluation\":\"2024-03-27T08:45:23.791195078Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicaSet\\\"}, \\\"replicaset\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicaset, namespace) group_left (owner_name) topk by (replicaset, namespace) (1, max by (replicaset, namespace, owner_name) (kube_replicaset_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deployment\"},\"health\":\"ok\",\"evaluationTime\":0.009879996,\"lastEvaluation\":\"2024-03-27T08:45:23.781310472Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"Job\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"job\"},\"health\":\"ok\",\"evaluationTime\":0.000706558,\"lastEvaluation\":\"2024-03-27T08:45:23.795465836Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"StatefulSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"statefulset\"},\"health\":\"ok\",\"evaluationTime\":0.001294074,\"lastEvaluation\":\"2024-03-27T08:45:23.794166953Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate\",\"query\":\"sum by (cluster, namespace, pod, container) (irate(container_cpu_usage_seconds_total{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"}[5m])) * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.06250654,\"lastEvaluation\":\"2024-03-27T08:45:23.077364252Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_memory_cache\",\"query\":\"container_memory_cache{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.064029051,\"lastEvaluation\":\"2024-03-27T08:45:23.262278578Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_memory_rss\",\"query\":\"container_memory_rss{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.061969574,\"lastEvaluation\":\"2024-03-27T08:45:23.200300598Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_memory_swap\",\"query\":\"container_memory_swap{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.07711624,\"lastEvaluation\":\"2024-03-27T08:45:23.326321211Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_memory_working_set_bytes\",\"query\":\"container_memory_working_set_bytes{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.060407376,\"lastEvaluation\":\"2024-03-27T08:45:23.139881172Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.275982231,\"lastEvaluation\":\"2024-03-27T08:45:23.076615453Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-scheduler.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.00030357,\"lastEvaluation\":\"2024-03-27T08:45:30.098262654Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000312908,\"lastEvaluation\":\"2024-03-27T08:45:30.095884435Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.00040136,\"lastEvaluation\":\"2024-03-27T08:45:30.0929236Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000301222,\"lastEvaluation\":\"2024-03-27T08:45:30.096201631Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000380069,\"lastEvaluation\":\"2024-03-27T08:45:30.093329985Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000242133,\"lastEvaluation\":\"2024-03-27T08:45:30.090885143Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.001751755,\"lastEvaluation\":\"2024-03-27T08:45:30.096506599Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.00216515,\"lastEvaluation\":\"2024-03-27T08:45:30.093715009Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000825024,\"lastEvaluation\":\"2024-03-27T08:45:30.091129252Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.008549868,\"lastEvaluation\":\"2024-03-27T08:45:30.090021401Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubelet.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.004432477,\"lastEvaluation\":\"2024-03-27T08:45:23.554710211Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.004497185,\"lastEvaluation\":\"2024-03-27T08:45:23.550206212Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001992705,\"lastEvaluation\":\"2024-03-27T08:45:23.546625429Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.005092881,\"lastEvaluation\":\"2024-03-27T08:45:23.546619375Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-apps\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeContainerWaiting\",\"query\":\"sum by (namespace, pod, container, cluster) (kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"pod/{{ $labels.pod }} in namespace {{ $labels.namespace }} on container {{ $labels.container}} has been in waiting state for longer than 1 hour.\",\"summary\":\"Pod container waiting longer than 1 hour\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002153012,\"lastEvaluation\":\"2024-03-27T08:45:47.355995377Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetMisScheduled\",\"query\":\"kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.\",\"summary\":\"DaemonSet pods are misscheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001475897,\"lastEvaluation\":\"2024-03-27T08:45:47.360560273Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetNotScheduled\",\"query\":\"kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} - kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.\",\"summary\":\"DaemonSet pods are not scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002405728,\"lastEvaluation\":\"2024-03-27T08:45:47.358151446Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetRolloutStuck\",\"query\":\"((kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != 0) or (kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} has not finished or progressed for at least 30 minutes.\",\"summary\":\"DaemonSet rollout is stuck.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.009939706,\"lastEvaluation\":\"2024-03-27T08:45:47.346052138Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentGenerationMismatch\",\"query\":\"kube_deployment_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_deployment_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.\",\"summary\":\"Deployment generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006676917,\"lastEvaluation\":\"2024-03-27T08:45:47.318742591Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentRolloutStuck\",\"query\":\"kube_deployment_status_condition{condition=\\\"Progressing\\\",job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",status=\\\"false\\\"} != 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Rollout of deployment {{ $labels.namespace }}/{{ $labels.deployment }} is not progressing for longer than 15 minutes.\",\"summary\":\"Deployment rollout is not progressing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002685861,\"lastEvaluation\":\"2024-03-27T08:45:47.325423617Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaMaxedOut\",\"query\":\"kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has been running at max replicas for longer than 15 minutes.\",\"summary\":\"HPA is running at max replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001748196,\"lastEvaluation\":\"2024-03-27T08:45:47.372211016Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaReplicasMismatch\",\"query\":\"(kube_horizontalpodautoscaler_status_desired_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_horizontalpodautoscaler_spec_min_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003c kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and changes(kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[15m]) == 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has not matched the desired number of replicas for longer than 15 minutes.\",\"summary\":\"HPA has not matched desired number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006123019,\"lastEvaluation\":\"2024-03-27T08:45:47.366084912Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobFailed\",\"query\":\"kube_job_failed{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeJobFailed.md\",\"summary\":\"Job failed to complete.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000962067,\"lastEvaluation\":\"2024-03-27T08:45:47.36511672Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobNotCompleted\",\"query\":\"time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} and kube_job_status_active{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0) \\u003e 43200\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than {{ \\\"43200\\\" | humanizeDuration }} to complete.\",\"summary\":\"Job did not complete in time\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003074286,\"lastEvaluation\":\"2024-03-27T08:45:47.362039128Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodCrashLooping\",\"query\":\"max_over_time(kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",reason=\\\"CrashLoopBackOff\\\"}[5m]) \\u003e= 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is in waiting state (reason: \\\"CrashLoopBackOff\\\").\",\"summary\":\"Pod is crash looping.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001448454,\"lastEvaluation\":\"2024-03-27T08:45:47.274928908Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodNotReady\",\"query\":\"sum by (namespace, pod, cluster) (max by (namespace, pod, cluster) (kube_pod_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Pending|Unknown\\\"} unless ignoring (phase) (kube_pod_status_unschedulable{job=\\\"kube-state-metrics\\\"} == 1)) * on (namespace, pod, cluster) group_left (owner_kind) topk by (namespace, pod, cluster) (1, max by (namespace, pod, owner_kind, cluster) (kube_pod_owner{owner_kind!=\\\"Job\\\"}))) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePodNotReady.md\",\"summary\":\"Pod has been in a non-ready state for more than 15 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.042353807,\"lastEvaluation\":\"2024-03-27T08:45:47.276380959Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetGenerationMismatch\",\"query\":\"kube_statefulset_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.\",\"summary\":\"StatefulSet generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003681367,\"lastEvaluation\":\"2024-03-27T08:45:47.333481829Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetReplicasMismatch\",\"query\":\"(kube_statefulset_status_replicas_ready{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[10m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.\",\"summary\":\"StatefulSet has not matched the expected number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.005360424,\"lastEvaluation\":\"2024-03-27T08:45:47.328115341Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetUpdateNotRolledOut\",\"query\":\"(max without (revision) (kube_statefulset_status_current_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} unless kube_statefulset_status_update_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) * (kube_statefulset_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.\",\"summary\":\"StatefulSet update has not been rolled out.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.008858762,\"lastEvaluation\":\"2024-03-27T08:45:47.337166375Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.099051569,\"lastEvaluation\":\"2024-03-27T08:45:47.27491246Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-resources\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeCPUOvercommit\",\"query\":\"sum by (cluster) (namespace_cpu:kube_pod_container_resource_requests:sum{job=\\\"kube-state-metrics\\\"}) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted CPU resource requests for Pods by {{ $value }} CPU shares and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted CPU resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000825829,\"lastEvaluation\":\"2024-03-27T08:45:25.808313079Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeMemoryOvercommit\",\"query\":\"sum by (cluster) (namespace_memory:kube_pod_container_resource_requests:sum) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted memory resource requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted memory resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002568247,\"lastEvaluation\":\"2024-03-27T08:45:25.810491086Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaAlmostFull\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) \\u003e 0.9 \\u003c 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota is going to be full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002669272,\"lastEvaluation\":\"2024-03-27T08:45:25.81306372Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaExceeded\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) \\u003e 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota has exceeded the limits.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002161789,\"lastEvaluation\":\"2024-03-27T08:45:25.817907741Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaFullyUsed\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota is fully used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002168344,\"lastEvaluation\":\"2024-03-27T08:45:25.815736285Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.006557557,\"lastEvaluation\":\"2024-03-27T08:45:25.80830289Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-storage\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeErrors\",\"query\":\"kube_persistentvolume_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Failed|Pending\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The persistent volume {{ $labels.persistentvolume }} has status {{ $labels.phase }}.\",\"summary\":\"PersistentVolume is having issues with provisioning.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00250918,\"lastEvaluation\":\"2024-03-27T08:45:35.315229019Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeFillingUp\",\"query\":\"(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_capacity_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.03 and kubelet_volume_stats_used_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage }} free.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeFillingUp.md\",\"summary\":\"PersistentVolume is filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003537372,\"lastEvaluation\":\"2024-03-27T08:45:35.286850559Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeFillingUp\",\"query\":\"(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_capacity_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.15 and kubelet_volume_stats_used_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 and predict_linear(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[6h], 4 * 24 * 3600) \\u003c 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to fill up within four days. Currently {{ $value | humanizePercentage }} is available.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeFillingUp.md\",\"summary\":\"PersistentVolume is filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.007874759,\"lastEvaluation\":\"2024-03-27T08:45:35.293892595Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeInodesFillingUp\",\"query\":\"(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_inodes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.03 and kubelet_volume_stats_inodes_used{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} only has {{ $value | humanizePercentage }} free inodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeInodesFillingUp.md\",\"summary\":\"PersistentVolumeInodes are filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.005711564,\"lastEvaluation\":\"2024-03-27T08:45:35.301770907Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeInodesFillingUp\",\"query\":\"(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_inodes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.15 and kubelet_volume_stats_inodes_used{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 and predict_linear(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[6h], 4 * 24 * 3600) \\u003c 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to run out of inodes within four days. Currently {{ $value | humanizePercentage }} of its inodes are free.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeInodesFillingUp.md\",\"summary\":\"PersistentVolumeInodes are filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.007740114,\"lastEvaluation\":\"2024-03-27T08:45:35.307485649Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.015490771,\"lastEvaluation\":\"2024-03-27T08:45:35.286836343Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeClientErrors\",\"query\":\"(sum by (cluster, instance, job, namespace) (rate(rest_client_requests_total{code=~\\\"5..\\\",job=\\\"apiserver\\\"}[5m])) / sum by (cluster, instance, job, namespace) (rate(rest_client_requests_total{job=\\\"apiserver\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ $value | humanizePercentage }} errors.'\",\"summary\":\"Kubernetes API server client is experiencing errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00285892,\"lastEvaluation\":\"2024-03-27T08:45:30.413499193Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00287564,\"lastEvaluation\":\"2024-03-27T08:45:30.413486607Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system-apiserver\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIDown\",\"query\":\"absent(up{job=\\\"apiserver\\\"} == 1)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeAPI has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeAPIDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000671169,\"lastEvaluation\":\"2024-03-27T08:45:23.68294632Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPITerminatedRequests\",\"query\":\"sum(rate(apiserver_request_terminations_total{job=\\\"apiserver\\\"}[10m])) / (sum(rate(apiserver_request_total{job=\\\"apiserver\\\"}[10m])) + sum(rate(apiserver_request_terminations_total{job=\\\"apiserver\\\"}[10m]))) \\u003e 0.2\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.\",\"summary\":\"The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.151050422,\"lastEvaluation\":\"2024-03-27T08:45:23.683623689Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAggregatedAPIDown\",\"query\":\"(1 - max by (name, namespace, cluster) (avg_over_time(aggregator_unavailable_apiservice{job=\\\"apiserver\\\"}[10m]))) * 100 \\u003c 85\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m.\",\"summary\":\"Kubernetes aggregated API is down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.0037326,\"lastEvaluation\":\"2024-03-27T08:45:23.671116978Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAggregatedAPIErrors\",\"query\":\"sum by (name, namespace, cluster) (increase(aggregator_unavailable_apiservice_total{job=\\\"apiserver\\\"}[10m])) \\u003e 4\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has reported errors. It has appeared unavailable {{ $value | humanize }} times averaged over the past 10m.\",\"summary\":\"Kubernetes aggregated API has reported errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00026527,\"lastEvaluation\":\"2024-03-27T08:45:23.670849456Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.164346519,\"lastEvaluation\":\"2024-03-27T08:45:23.670333939Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system-kubelet\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeNodeNotReady\",\"query\":\"kube_node_status_condition{condition=\\\"Ready\\\",job=\\\"kube-state-metrics\\\",status=\\\"true\\\"} == 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.node }} has been unready for more than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeNodeNotReady.md\",\"summary\":\"Node is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00042682,\"lastEvaluation\":\"2024-03-27T08:45:39.335864676Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeNodeReadinessFlapping\",\"query\":\"sum by (cluster, node) (changes(kube_node_status_condition{condition=\\\"Ready\\\",job=\\\"kube-state-metrics\\\",status=\\\"true\\\"}[15m])) \\u003e 2\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The readiness status of node {{ $labels.node }} has changed {{ $value }} times in the last 15 minutes.\",\"summary\":\"Node readiness status is flapping.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000460336,\"lastEvaluation\":\"2024-03-27T08:45:39.359519004Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeNodeUnreachable\",\"query\":\"(kube_node_spec_taint{effect=\\\"NoSchedule\\\",job=\\\"kube-state-metrics\\\",key=\\\"node.kubernetes.io/unreachable\\\"} unless ignoring (key, value) kube_node_spec_taint{job=\\\"kube-state-metrics\\\",key=~\\\"ToBeDeletedByClusterAutoscaler|cloud.google.com/impending-node-termination|aws-node-termination-handler/spot-itn\\\"}) == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.node }} is unreachable and some workloads may be rescheduled.\",\"summary\":\"Node is unreachable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000731907,\"lastEvaluation\":\"2024-03-27T08:45:39.336294826Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletClientCertificateRenewalErrors\",\"query\":\"increase(kubelet_certificate_manager_client_expiration_renew_errors[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet on node {{ $labels.node }} has failed to renew its client certificate ({{ $value | humanize }} errors in the last 5 minutes).\",\"summary\":\"Kubelet has failed to renew its client certificate.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00032062,\"lastEvaluation\":\"2024-03-27T08:45:39.367174869Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletDown\",\"query\":\"absent(up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Kubelet has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeletDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000347164,\"lastEvaluation\":\"2024-03-27T08:45:39.367843204Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletPlegDurationHigh\",\"query\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile{quantile=\\\"0.99\\\"} \\u003e= 10\",\"duration\":300,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Kubelet Pod Lifecycle Event Generator has a 99th percentile duration of {{ $value }} seconds on node {{ $labels.node }}.\",\"summary\":\"Kubelet Pod Lifecycle Event Generator is taking too long to relist.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000309928,\"lastEvaluation\":\"2024-03-27T08:45:39.359982319Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletPodStartUpLatencyHigh\",\"query\":\"histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m]))) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} \\u003e 60\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on node {{ $labels.node }}.\",\"summary\":\"Kubelet Pod startup latency is too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006876424,\"lastEvaluation\":\"2024-03-27T08:45:39.360295119Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletServerCertificateRenewalErrors\",\"query\":\"increase(kubelet_server_expiration_renew_errors[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet on node {{ $labels.node }} has failed to renew its server certificate ({{ $value | humanize }} errors in the last 5 minutes).\",\"summary\":\"Kubelet has failed to renew its server certificate.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000341709,\"lastEvaluation\":\"2024-03-27T08:45:39.367498524Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletTooManyPods\",\"query\":\"count by (cluster, node) ((kube_pod_status_phase{job=\\\"kube-state-metrics\\\",phase=\\\"Running\\\"} == 1) * on (instance, pod, namespace, cluster) group_left (node) topk by (instance, pod, namespace, cluster) (1, kube_pod_info{job=\\\"kube-state-metrics\\\"})) / max by (cluster, node) (kube_node_status_capacity{job=\\\"kube-state-metrics\\\",resource=\\\"pods\\\"} != 1) \\u003e 0.95\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage }} of its Pod capacity.\",\"summary\":\"Kubelet is running at capacity.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.022481053,\"lastEvaluation\":\"2024-03-27T08:45:39.337032795Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.032344756,\"lastEvaluation\":\"2024-03-27T08:45:39.335849581Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"name\":\":node_memory_MemAvailable_bytes:sum\",\"query\":\"sum by (cluster) (node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"} or (node_memory_Buffers_bytes{job=\\\"node-exporter\\\"} + node_memory_Cached_bytes{job=\\\"node-exporter\\\"} + node_memory_MemFree_bytes{job=\\\"node-exporter\\\"} + node_memory_Slab_bytes{job=\\\"node-exporter\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001237524,\"lastEvaluation\":\"2024-03-27T08:45:31.14650583Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:ratio_rate5m\",\"query\":\"avg by (cluster) (node:node_cpu_utilization:ratio_rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00026181,\"lastEvaluation\":\"2024-03-27T08:45:31.15275632Z\",\"type\":\"recording\"},{\"name\":\"node:node_cpu_utilization:ratio_rate5m\",\"query\":\"avg by (cluster, node) (sum without (mode) (rate(node_cpu_seconds_total{job=\\\"node-exporter\\\",mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00500557,\"lastEvaluation\":\"2024-03-27T08:45:31.147747273Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod:kube_pod_info:\",\"query\":\"topk by (cluster, namespace, pod) (1, max by (cluster, node, namespace, pod) (label_replace(kube_pod_info{job=\\\"kube-state-metrics\\\",node!=\\\"\\\"}, \\\"pod\\\", \\\"$1\\\", \\\"pod\\\", \\\"(.*)\\\")))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009058184,\"lastEvaluation\":\"2024-03-27T08:45:31.131886749Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.021295429,\"lastEvaluation\":\"2024-03-27T08:45:31.131727245Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-exporter\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-node-exporter-rules-08280c24-7295-4eea-b6e8-23a031271619.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeClockNotSynchronising\",\"query\":\"min_over_time(node_timex_sync_status{job=\\\"node-exporter\\\"}[5m]) == 0 and node_timex_maxerror_seconds{job=\\\"node-exporter\\\"} \\u003e= 16\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Clock at {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeClockNotSynchronising.md\",\"summary\":\"Clock not synchronising.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000699412,\"lastEvaluation\":\"2024-03-27T08:45:30.322928976Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeClockSkewDetected\",\"query\":\"(node_timex_offset_seconds{job=\\\"node-exporter\\\"} \\u003e 0.05 and deriv(node_timex_offset_seconds{job=\\\"node-exporter\\\"}[5m]) \\u003e= 0) or (node_timex_offset_seconds{job=\\\"node-exporter\\\"} \\u003c -0.05 and deriv(node_timex_offset_seconds{job=\\\"node-exporter\\\"}[5m]) \\u003c= 0)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Clock at {{ $labels.instance }} is out of sync by more than 0.05s. Ensure NTP is configured correctly on this host.\",\"summary\":\"Clock skew detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001305509,\"lastEvaluation\":\"2024-03-27T08:45:30.321619454Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFileDescriptorLimit\",\"query\":\"(node_filefd_allocated{job=\\\"node-exporter\\\"} * 100 / node_filefd_maximum{job=\\\"node-exporter\\\"} \\u003e 90)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"File descriptors limit at {{ $labels.instance }} is currently at {{ printf \\\"%.2f\\\" $value }}%.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFileDescriptorLimit.md\",\"summary\":\"Kernel is predicted to exhaust file descriptors limit soon.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000545824,\"lastEvaluation\":\"2024-03-27T08:45:30.325443648Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFileDescriptorLimit\",\"query\":\"(node_filefd_allocated{job=\\\"node-exporter\\\"} * 100 / node_filefd_maximum{job=\\\"node-exporter\\\"} \\u003e 70)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"File descriptors limit at {{ $labels.instance }} is currently at {{ printf \\\"%.2f\\\" $value }}%.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFileDescriptorLimit.md\",\"summary\":\"Kernel is predicted to exhaust file descriptors limit soon.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000550624,\"lastEvaluation\":\"2024-03-27T08:45:30.324890156Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfFiles\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 3 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfFiles.md\",\"summary\":\"Filesystem has less than 3% inodes left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006735677,\"lastEvaluation\":\"2024-03-27T08:45:30.309744125Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfFiles\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 5 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfFiles.md\",\"summary\":\"Filesystem has less than 5% inodes left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00620677,\"lastEvaluation\":\"2024-03-27T08:45:30.303532876Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfSpace\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 3 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfSpace.md\",\"summary\":\"Filesystem has less than 3% space left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.005772252,\"lastEvaluation\":\"2024-03-27T08:45:30.263885257Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfSpace\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 5 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfSpace.md\",\"summary\":\"Filesystem has less than 5% space left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.005903321,\"lastEvaluation\":\"2024-03-27T08:45:30.257978252Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemFilesFillingUp\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 20 and predict_linear(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 4 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left and is filling up fast.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemFilesFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of inodes within the next 4 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.016376743,\"lastEvaluation\":\"2024-03-27T08:45:30.287148843Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemFilesFillingUp\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 40 and predict_linear(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 24 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left and is filling up.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemFilesFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of inodes within the next 24 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.017481587,\"lastEvaluation\":\"2024-03-27T08:45:30.269661224Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemSpaceFillingUp\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 10 and predict_linear(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 4 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left and is filling up fast.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemSpaceFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of space within the next 4 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.016284432,\"lastEvaluation\":\"2024-03-27T08:45:30.241690107Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemSpaceFillingUp\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 15 and predict_linear(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 24 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left and is filling up.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemSpaceFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of space within the next 24 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.013148163,\"lastEvaluation\":\"2024-03-27T08:45:30.225036187Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeHighNumberConntrackEntriesUsed\",\"query\":\"(node_nf_conntrack_entries{job=\\\"node-exporter\\\"} / node_nf_conntrack_entries_limit) \\u003e 0.75\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of conntrack entries are used.\",\"summary\":\"Number of conntrack are getting close to the limit.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000596718,\"lastEvaluation\":\"2024-03-27T08:45:30.320669454Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeMemoryMajorPagesFaults\",\"query\":\"rate(node_vmstat_pgmajfault{job=\\\"node-exporter\\\"}[5m]) \\u003e 500\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Memory major pages are occurring at very high rate at {{ $labels.instance }}, 500 major page faults per second for the last 15 minutes, is currently at {{ printf \\\"%.2f\\\" $value }}.\\nPlease check that there is enough memory available at this instance.\\n\",\"summary\":\"Memory major page faults are occurring at very high rate.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000364874,\"lastEvaluation\":\"2024-03-27T08:45:30.327305486Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeNetworkReceiveErrs\",\"query\":\"rate(node_network_receive_errs_total{job=\\\"node-exporter\\\"}[2m]) / rate(node_network_receive_packets_total{job=\\\"node-exporter\\\"}[2m]) \\u003e 0.01\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \\\"%.0f\\\" $value }} receive errors in the last two minutes.\",\"summary\":\"Network interface is reporting many receive errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002129078,\"lastEvaluation\":\"2024-03-27T08:45:30.316483874Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeNetworkTransmitErrs\",\"query\":\"rate(node_network_transmit_errs_total{job=\\\"node-exporter\\\"}[2m]) / rate(node_network_transmit_packets_total{job=\\\"node-exporter\\\"}[2m]) \\u003e 0.01\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \\\"%.0f\\\" $value }} transmit errors in the last two minutes.\",\"summary\":\"Network interface is reporting many transmit errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002048971,\"lastEvaluation\":\"2024-03-27T08:45:30.318617441Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeRAIDDegraded\",\"query\":\"node_md_disks_required{device=~\\\"mmcblk.p.+|nvme.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\\\",job=\\\"node-exporter\\\"} - ignoring (state) (node_md_disks{device=~\\\"mmcblk.p.+|nvme.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\\\",job=\\\"node-exporter\\\",state=\\\"active\\\"}) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"RAID array '{{ $labels.device }}' at {{ $labels.instance }} is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeRAIDDegraded.md\",\"summary\":\"RAID Array is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000809137,\"lastEvaluation\":\"2024-03-27T08:45:30.323631648Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeRAIDDiskFailure\",\"query\":\"node_md_disks{device=~\\\"mmcblk.p.+|nvme.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\\\",job=\\\"node-exporter\\\",state=\\\"failed\\\"} \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"At least one device in RAID array at {{ $labels.instance }} failed. Array '{{ $labels.device }}' needs attention and possibly a disk swap.\",\"summary\":\"Failed device in RAID array.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000443134,\"lastEvaluation\":\"2024-03-27T08:45:30.324443953Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeSystemSaturation\",\"query\":\"node_load1{job=\\\"node-exporter\\\"} / count without (cpu, mode) (node_cpu_seconds_total{job=\\\"node-exporter\\\",mode=\\\"idle\\\"}) \\u003e 2\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"System load per core at {{ $labels.instance }} has been above 2 for the last 15 minutes, is currently at {{ printf \\\"%.2f\\\" $value }}.\\nThis might indicate this instance resources saturation and can cause it becoming unresponsive.\\n\",\"summary\":\"System saturated, load per core is very high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001310056,\"lastEvaluation\":\"2024-03-27T08:45:30.325992352Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeSystemdServiceFailed\",\"query\":\"node_systemd_unit_state{job=\\\"node-exporter\\\",state=\\\"failed\\\"} == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Systemd service {{ $labels.name }} has entered failed state at {{ $labels.instance }}\",\"summary\":\"Systemd service has entered failed state.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000277385,\"lastEvaluation\":\"2024-03-27T08:45:30.327673649Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeTextFileCollectorScrapeError\",\"query\":\"node_textfile_scrape_error{job=\\\"node-exporter\\\"} == 1\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Node Exporter text file collector on {{ $labels.instance }} failed to scrape.\",\"summary\":\"Node Exporter text file collector failed to scrape.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000345288,\"lastEvaluation\":\"2024-03-27T08:45:30.321270129Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.103072468,\"lastEvaluation\":\"2024-03-27T08:45:30.224883615Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-exporter.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-node-exporter-rules-08280c24-7295-4eea-b6e8-23a031271619.yaml\",\"rules\":[{\"name\":\"instance:node_cpu_utilisation:rate1m\",\"query\":\"1 - avg without (cpu) (sum without (mode) (rate(node_cpu_seconds_total{job=\\\"node-exporter\\\",mode=~\\\"idle|iowait|steal\\\"}[1m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002833082,\"lastEvaluation\":\"2024-03-27T08:45:26.199353844Z\",\"type\":\"recording\"},{\"name\":\"instance:node_load1_per_cpu:ratio\",\"query\":\"(node_load1{job=\\\"node-exporter\\\"} / instance:node_num_cpu:sum{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000724354,\"lastEvaluation\":\"2024-03-27T08:45:26.202821014Z\",\"type\":\"recording\"},{\"name\":\"instance:node_memory_utilisation:ratio\",\"query\":\"1 - ((node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"} or (node_memory_Buffers_bytes{job=\\\"node-exporter\\\"} + node_memory_Cached_bytes{job=\\\"node-exporter\\\"} + node_memory_MemFree_bytes{job=\\\"node-exporter\\\"} + node_memory_Slab_bytes{job=\\\"node-exporter\\\"})) / node_memory_MemTotal_bytes{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001689371,\"lastEvaluation\":\"2024-03-27T08:45:26.203549952Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_bytes_excluding_lo:rate1m\",\"query\":\"sum without (device) (rate(node_network_receive_bytes_total{device!=\\\"lo\\\",job=\\\"node-exporter\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00114913,\"lastEvaluation\":\"2024-03-27T08:45:26.213320257Z\",\"type\":\"recording\"},{\"na{\"status\":\"success\",\"data\":{\"groups\":[{\"name\":\"CloudCredentialOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-c2ad383d-f0d6-40eb-947b-40f88973bc7a.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorDeprovisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsDeprovisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be cleaned up\",\"summary\":\"One or more CredentialsRequest CRs are unable to be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000131974,\"lastEvaluation\":\"2024-03-27T08:45:25.248120814Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorInsufficientCloudCreds\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"InsufficientCloudCreds\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.\",\"message\":\"Cluster's cloud credentials insufficient for minting or passthrough\",\"summary\":\"Problem with the available platform credentials.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000127798,\"lastEvaluation\":\"2024-03-27T08:45:25.248255358Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorProvisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsProvisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be fulfilled\",\"summary\":\"One or more CredentialsRequest CRs are unable to be processed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000126631,\"lastEvaluation\":\"2024-03-27T08:45:25.247992357Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorStaleCredentials\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"StaleCredentials\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.\",\"message\":\"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).\",\"summary\":\"One or more CredentialsRequest CRs are stale and should be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00012323,\"lastEvaluation\":\"2024-03-27T08:45:25.248384928Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorTargetNamespaceMissing\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"MissingTargetNamespace\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.\",\"message\":\"CredentialsRequest(s) pointing to non-existent namespace\",\"summary\":\"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000238978,\"lastEvaluation\":\"2024-03-27T08:45:25.247751221Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001258071,\"lastEvaluation\":\"2024-03-27T08:45:25.247200671Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-machine-approver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-fdba86e4-24de-4c05-9e26-5f58dc7abb5c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineApproverMaxPendingCSRsReached\",\"query\":\"mapi_current_pending_csr \\u003e mapi_max_pending_csr\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of pending CertificateSigningRequests has exceeded the\\nmaximum threshold (current number of machine + 100). Check the\\npending CSRs to determine which machines need approval, also check\\nthat the nodelink controller is running in the openshift-machine-api\\nnamespace.\\n\",\"summary\":\"max pending CSRs threshold reached.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00032469,\"lastEvaluation\":\"2024-03-27T08:45:36.292140978Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000339832,\"lastEvaluation\":\"2024-03-27T08:45:36.292131304Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-tuning-operator.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-6bc0735b-af85-4938-91c4-907e725cd4a8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NTODegraded\",\"query\":\"nto_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Node Tuning Operator is degraded. Review the \\\"node-tuning\\\" ClusterOperator object for further details.\",\"summary\":\"The Node Tuning Operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000204003,\"lastEvaluation\":\"2024-03-27T08:45:28.787760075Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NTOPodsNotReady\",\"query\":\"kube_pod_status_ready{condition=\\\"true\\\",namespace=\\\"openshift-cluster-node-tuning-operator\\\"} == 0\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.pod }} is not ready.\\nReview the \\\"Event\\\" objects in \\\"openshift-cluster-node-tuning-operator\\\" namespace for further details.\\n\",\"summary\":\"Pod {{ $labels.pod }} is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000599731,\"lastEvaluation\":\"2024-03-27T08:45:28.787143266Z\",\"type\":\"alerting\"},{\"name\":\"nto_custom_profiles:count\",\"query\":\"count by (_id) (nto_profile_calculated_total{profile!~\\\"openshift\\\",profile!~\\\"openshift-control-plane\\\",profile!~\\\"openshift-node\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000273454,\"lastEvaluation\":\"2024-03-27T08:45:28.788013798Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.00108666,\"lastEvaluation\":\"2024-03-27T08:45:28.787079593Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"SamplesOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7d6b51ae-913b-48fc-bbed-8261aa1794e5.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SamplesDegraded\",\"query\":\"openshift_samples_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples could not be deployed and the operator is degraded. Review the \\\"openshift-samples\\\" ClusterOperator object for further details.\\n\",\"summary\":\"Samples operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000306286,\"lastEvaluation\":\"2024-03-27T08:45:39.16708304Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesImagestreamImportFailing\",\"query\":\"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \\u003e sum(openshift_samples_failed_imagestream_import_info)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports.  You can look at the \\\"openshift-samples\\\"\\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is detecting problems with imagestream image imports\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001259634,\"lastEvaluation\":\"2024-03-27T08:45:39.165818393Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesInvalidConfig\",\"query\":\"openshift_samples_invalidconfig_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator has been given an invalid configuration.\\n\",\"summary\":\"Samples operator Invalid configuration\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00028186,\"lastEvaluation\":\"2024-03-27T08:45:39.167393914Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingSecret\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_secret\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator cannot find the samples pull secret in the openshift namespace.\\n\",\"summary\":\"Samples operator is not able to find secret\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000284289,\"lastEvaluation\":\"2024-03-27T08:45:39.167680061Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingTBRCredential\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_tbr_credential\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\\n\",\"summary\":\"Samples operator is not able to find the credentials for registry\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000265952,\"lastEvaluation\":\"2024-03-27T08:45:39.167968285Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesRetriesMissingOnImagestreamImportFailing\",\"query\":\"sum(openshift_samples_failed_imagestream_import_info) \\u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\\nimports are not occurring.  Contact support.  You can look at the \\\"openshift-samples\\\" ClusterOperator object\\nfor details. Most likely there are issues with the external image registry hosting the images that need to\\nbe investigated.  The list of ImageStreams that have failing imports are:\\n{{ range query \\\"openshift_samples_failed_imagestream_import_info \\u003e 0\\\" }}\\n  {{ .Labels.name }}\\n{{ end }}\\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is having problems with imagestream imports and its retries.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000761995,\"lastEvaluation\":\"2024-03-27T08:45:39.164894853Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesTBRInaccessibleOnBoot\",\"query\":\"openshift_samples_tbr_inaccessible_info == 1\",\"duration\":172800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"One of two situations has occurred.  Either\\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\\nassociated with each image stream tag from the samples catalog is\\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\\nassist the mirroring process.\\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\\n\",\"summary\":\"Samples operator is not able to access the registry on boot\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00030891,\"lastEvaluation\":\"2024-03-27T08:45:39.168238101Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.0018249,\"lastEvaluation\":\"2024-03-27T08:45:39.164880253Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"default-storage-classes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-2fde48fd-01e3-4ab0-8d3c-aab39c0159d4.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MultipleDefaultStorageClasses\",\"query\":\"max_over_time(default_storage_class_count[5m]) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster storage operator monitors all storage classes configured in the cluster\\nand checks there is not more than one default StorageClass configured.\\n\",\"message\":\"StorageClass count check is failing (there should not be more than one default StorageClass)\",\"summary\":\"More than one default StorageClass detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000499849,\"lastEvaluation\":\"2024-03-27T08:45:24.690196242Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000318698,\"lastEvaluation\":\"2024-03-27T08:45:24.689472838Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-2fde48fd-01e3-4ab0-8d3c-aab39c0159d4.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodStartupStorageOperationsFailing\",\"query\":\"increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status!=\\\"success\\\"}[5m]) \\u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status=\\\"success\\\"}[5m]) == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Failing storage operation \\\"{{ $labels.operation_name }}\\\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\\nfrom starting for past 5 minutes.\\nPlease investigate Pods that are \\\"ContainerCreating\\\" on the node: \\\"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\\\".\\nEvents of the Pods should contain exact error message: \\\"oc describe pod -n \\u003cpod namespace\\u003e \\u003cpod name\\u003e\\\".\\n\",\"summary\":\"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006817402,\"lastEvaluation\":\"2024-03-27T08:45:29.13735178Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003636162,\"lastEvaluation\":\"2024-03-27T08:45:29.136332985Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-operators\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-386af8f5-02f3-497a-8b16-8c3119059b4f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ClusterNotUpgradeable\",\"query\":\"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\\\"Upgradeable\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"Upgradeable\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0) (ne (len (label \\\"reason\\\" $value)) 0) }}{{label \\\"reason\\\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000491204,\"lastEvaluation\":\"2024-03-27T08:45:25.151652979Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDegraded\",\"query\":\"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\\\"Degraded\\\",job=\\\"cluster-version-operator\\\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\\\"cluster-version-operator\\\"})) == 1)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has been degraded for 30 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002183759,\"lastEvaluation\":\"2024-03-27T08:45:25.153279268Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDown\",\"query\":\"max by (namespace, name, reason) (cluster_operator_up{job=\\\"cluster-version-operator\\\"} == 0)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has not been available for 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001126798,\"lastEvaluation\":\"2024-03-27T08:45:25.152148597Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorFlapping\",\"query\":\"max by (namespace, name) (changes(cluster_operator_up{job=\\\"cluster-version-operator\\\"}[2m]) \\u003e 2)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator up status is changing often.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000871143,\"lastEvaluation\":\"2024-03-27T08:45:25.155466915Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.004698528,\"lastEvaluation\":\"2024-03-27T08:45:25.151643037Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-386af8f5-02f3-497a-8b16-8c3119059b4f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotRetrieveUpdates\",\"query\":\"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \\u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\\\"RetrievedUpdates\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\",reason!=\\\"NoChannel\\\"}))\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"RetrievedUpdates\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0)}}{{label \\\"reason\\\" $value}} {{end}}{{end}}{{end}}. {{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} For more information refer to {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/.{{ end }}{{ end }}\",\"summary\":\"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000573484,\"lastEvaluation\":\"2024-03-27T08:45:38.443436604Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterReleaseNotAccepted\",\"query\":\"max by (namespace, name, reason) (cluster_operator_conditions{condition=\\\"ReleaseAccepted\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"The desired cluster release has not been accepted for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000316833,\"lastEvaluation\":\"2024-03-27T08:45:38.444237058Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterVersionOperatorDown\",\"query\":\"absent(up{job=\\\"cluster-version-operator\\\"} == 1)\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-cluster-version\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} For more information refer to {{ label \\\"url\\\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}\",\"summary\":\"Cluster version operator has disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000356209,\"lastEvaluation\":\"2024-03-27T08:45:38.443076825Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UpdateAvailable\",\"query\":\"sum by (channel, namespace, upstream) (cluster_version_available_updates) \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Your upstream update recommendation service recommends you update your cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000218598,\"lastEvaluation\":\"2024-03-27T08:45:38.444013933Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001493469,\"lastEvaluation\":\"2024-03-27T08:45:38.443066052Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift/console-operator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-ac9f320d-b477-4dca-b2d1-9c630c99bd96.yaml\",\"rules\":[{\"name\":\"cluster:console_auth_login_failures_total:sum\",\"query\":\"sum by (reason) (console_auth_login_failures_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000182904,\"lastEvaluation\":\"2024-03-27T08:45:23.269220966Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_requests_total:sum\",\"query\":\"sum(console_auth_login_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000252171,\"lastEvaluation\":\"2024-03-27T08:45:23.268705929Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_successes_total:sum\",\"query\":\"sum by (role) (console_auth_login_successes_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0002551,\"lastEvaluation\":\"2024-03-27T08:45:23.268960858Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_logout_requests_total:sum\",\"query\":\"sum by (reason) (console_auth_logout_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000178597,\"lastEvaluation\":\"2024-03-27T08:45:23.269408522Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_customization_perspectives_info:max\",\"query\":\"max by (name, state) (console_customization_perspectives_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000129358,\"lastEvaluation\":\"2024-03-27T08:45:23.269887504Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_plugins_info:max\",\"query\":\"max by (name, state) (console_plugins_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000175232,\"lastEvaluation\":\"2024-03-27T08:45:23.269709797Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_usage_users:max\",\"query\":\"max by (role) (console_usage_users)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000117567,\"lastEvaluation\":\"2024-03-27T08:45:23.269589819Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002834942,\"lastEvaluation\":\"2024-03-27T08:45:23.25701976Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-dns.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-9c32b614-9906-4eb1-a991-5265655eef6f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CoreDNSErrorsHigh\",\"query\":\"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\\\"SERVFAIL\\\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.\",\"summary\":\"CoreDNS serverfail\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001005996,\"lastEvaluation\":\"2024-03-27T08:45:23.533650042Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSHealthCheckSlow\",\"query\":\"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \\u003e 10\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})\",\"summary\":\"CoreDNS health checks\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000861207,\"lastEvaluation\":\"2024-03-27T08:45:23.531862846Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSPanicking\",\"query\":\"increase(coredns_panics_total[10m]) \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}\",\"summary\":\"CoreDNS panic\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000390121,\"lastEvaluation\":\"2024-03-27T08:45:23.531469541Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001801079,\"lastEvaluation\":\"2024-03-27T08:45:23.531457665Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"etcd\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4b464ec4-d807-4a1f-9b84-bc7672f9c9b8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdDatabaseHighFragmentationRatio\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \\u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \\u003e 1.048576e+08\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.\",\"runbook_url\":\"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation\",\"summary\":\"etcd database size in use is less than 50% of the actual allocated storage.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000649654,\"lastEvaluation\":\"2024-03-27T08:45:46.524554715Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdDatabaseQuotaLowSpace\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \\u003e 95\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md\",\"summary\":\"etcd cluster database is running full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000488487,\"lastEvaluation\":\"2024-03-27T08:45:46.523502322Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdExcessiveDatabaseGrowth\",\"query\":\"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \\u003e etcd_server_quota_backend_bytes\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.\",\"summary\":\"etcd cluster database growing very fast.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558184,\"lastEvaluation\":\"2024-03-27T08:45:46.52399361Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighCommitDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.25\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile commit durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001379291,\"lastEvaluation\":\"2024-03-27T08:45:46.522119761Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001279039,\"lastEvaluation\":\"2024-03-27T08:45:46.520837777Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.5\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001492452,\"lastEvaluation\":\"2024-03-27T08:45:46.519342458Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedProposals\",\"query\":\"rate(etcd_server_proposals_failed_total{job=~\\\".*etcd.*\\\"}[15m]) \\u003e 5\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of proposal failures.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000386351,\"lastEvaluation\":\"2024-03-27T08:45:46.518953184Z\",\"type\":\"alerting\"},{\"state\":\"pending\",\"name\":\"etcdMemberCommunicationSlow\",\"query\":\"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.15\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster member communication is slow.\"},\"alerts\":[{\"labels\":{\"To\":\"4e1f58f3f6f92028\",\"alertname\":\"etcdMemberCommunicationSlow\",\"endpoint\":\"etcd-metrics\",\"instance\":\"192.168.131.12:9979\",\"job\":\"etcd\",\"namespace\":\"openshift-etcd\",\"pod\":\"etcd-p-syd04-4xtgr-master-0\",\"service\":\"etcd\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"etcd\\\": member communication with 4e1f58f3f6f92028 is taking 0.1863680000000001s on etcd instance 192.168.131.12:9979.\",\"summary\":\"etcd cluster member communication is slow.\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T08:41:46.512890203Z\",\"value\":\"1.863680000000001e-01\",\"partialResponseStrategy\":\"WARN\"},{\"labels\":{\"To\":\"fc0c538482edf5c3\",\"alertname\":\"etcdMemberCommunicationSlow\",\"endpoint\":\"etcd-metrics\",\"instance\":\"192.168.131.12:9979\",\"job\":\"etcd\",\"namespace\":\"openshift-etcd\",\"pod\":\"etcd-p-syd04-4xtgr-master-0\",\"service\":\"etcd\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"etcd\\\": member communication with fc0c538482edf5c3 is taking 0.1863680000000001s on etcd instance 192.168.131.12:9979.\",\"summary\":\"etcd cluster member communication is slow.\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T08:41:46.512890203Z\",\"value\":\"1.863680000000001e-01\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.003182665,\"lastEvaluation\":\"2024-03-27T08:45:46.515765726Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMembersDown\",\"query\":\"max without (endpoint) (sum without (instance) (up{job=~\\\".*etcd.*\\\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\\\".*etcd.*\\\"}[2m])) \\u003e 0.01)) \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": members are down ({{ $value }}).\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md\",\"summary\":\"etcd cluster members are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000991807,\"lastEvaluation\":\"2024-03-27T08:45:46.514499856Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdNoLeader\",\"query\":\"etcd_server_has_leader{job=~\\\".*etcd.*\\\"} == 0\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member {{ $labels.instance }} has no leader.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md\",\"summary\":\"etcd cluster has no leader.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000267827,\"lastEvaluation\":\"2024-03-27T08:45:46.515495073Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.010721875,\"lastEvaluation\":\"2024-03-27T08:45:46.514487751Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4b464ec4-d807-4a1f-9b84-bc7672f9c9b8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdGRPCRequestsSlow\",\"query\":\"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\\\"Defragment\\\",grpc_type=\\\"unary\\\",job=\\\"etcd\\\"}[10m]))) \\u003e 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md\",\"summary\":\"etcd grpc requests are slow\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.05584847,\"lastEvaluation\":\"2024-03-27T08:45:44.2668134Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 50\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.0962472,\"lastEvaluation\":\"2024-03-27T08:45:44.403877779Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 10\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.081203462,\"lastEvaluation\":\"2024-03-27T08:45:44.322669717Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfLeaderChanges\",\"query\":\"avg(changes(etcd_server_is_leader[10m])) \\u003e 5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.\",\"summary\":\"etcd cluster has high number of leader changes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000477891,\"lastEvaluation\":\"2024-03-27T08:45:44.500131619Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdInsufficientMembers\",\"query\":\"sum without (instance, pod) (up{job=\\\"etcd\\\"} == bool 1 and etcd_server_has_leader{job=\\\"etcd\\\"} == bool 1) \\u003c ((count without (instance, pod) (up{job=\\\"etcd\\\"}) + 1) / 2)\",\"duration\":180,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md\",\"summary\":\"etcd is reporting that a majority of instances are unavailable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000935162,\"lastEvaluation\":\"2024-03-27T08:45:44.500613429Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.111003446,\"lastEvaluation\":\"2024-03-27T08:45:44.26613945Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imageregistry.operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-34ed4fed-6db8-4d39-91ec-818754bb9a90.yaml\",\"rules\":[{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.Create\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000387633,\"lastEvaluation\":\"2024-03-27T08:45:26.210475488Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.ServeBlob\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00054476,\"lastEvaluation\":\"2024-03-27T08:45:26.209927518Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Get\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000360746,\"lastEvaluation\":\"2024-03-27T08:45:26.210866957Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Put\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000492941,\"lastEvaluation\":\"2024-03-27T08:45:26.211488715Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002325564,\"lastEvaluation\":\"2024-03-27T08:45:26.209661546Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imagestreams.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-3466f2a1-cd9d-4fb9-bc77-283ea39b85cb.yaml\",\"rules\":[{\"name\":\"imageregistry:imagestreamtags_count:sum\",\"query\":\"sum by (location, source) (image_registry_image_stream_tags_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000493705,\"lastEvaluation\":\"2024-03-27T08:45:40.762214503Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000514659,\"lastEvaluation\":\"2024-03-27T08:45:40.762198093Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress-to-route-controller.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-73a44dd8-1441-4959-8340-21724ebac1eb.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"IngressWithoutClassName\",\"query\":\"openshift_ingress_to_route_controller_ingress_without_class_name == 1\",\"duration\":86400,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"alerts\":[{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.129.0.56:8443\",\"job\":\"route-controller-manager\",\"name\":\"test\",\"namespace\":\"openshift-route-controller-manager\",\"pod\":\"route-controller-manager-747675d674-79hrk\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress test is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T08:26:43.136557208Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"},{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.129.0.56:8443\",\"job\":\"route-controller-manager\",\"name\":\"e2e-example-inghk797\",\"namespace\":\"openshift-route-controller-manager\",\"pod\":\"route-controller-manager-747675d674-79hrk\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress e2e-example-inghk797 is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T07:43:43.136557208Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"},{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.129.0.56:8443\",\"job\":\"route-controller-manager\",\"name\":\"e2e-example-ing7wmx9\",\"namespace\":\"openshift-route-controller-manager\",\"pod\":\"route-controller-manager-747675d674-79hrk\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress e2e-example-ing7wmx9 is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T07:43:43.136557208Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"},{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.129.0.56:8443\",\"job\":\"route-controller-manager\",\"name\":\"e2e-example-ingbdpl5\",\"namespace\":\"openshift-route-controller-manager\",\"pod\":\"route-controller-manager-747675d674-79hrk\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress e2e-example-ingbdpl5 is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-03-27T07:43:43.136557208Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000964852,\"lastEvaluation\":\"2024-03-27T08:45:43.138560461Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UnmanagedRoutes\",\"query\":\"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is a Route owned by an unmanaged Ingress.\",\"message\":\"Route {{ $labels.name }} is owned by an unmanaged Ingress.\",\"summary\":\"Route owned by an Ingress no longer managed\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002014366,\"lastEvaluation\":\"2024-03-27T08:45:43.140545507Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.004202031,\"lastEvaluation\":\"2024-03-27T08:45:43.1383644Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-73a44dd8-1441-4959-8340-21724ebac1eb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HAProxyDown\",\"query\":\"haproxy_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"This alert fires when metrics report that HAProxy is down.\",\"message\":\"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}\",\"summary\":\"HAProxy is down\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000134214,\"lastEvaluation\":\"2024-03-27T08:45:28.211383973Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HAProxyReloadFail\",\"query\":\"template_router_reload_failure == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.\",\"message\":\"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes\",\"summary\":\"HAProxy reload failure\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000254204,\"lastEvaluation\":\"2024-03-27T08:45:28.211127691Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerDegraded\",\"query\":\"ingress_controller_conditions{condition=\\\"Degraded\\\"} == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController status is degraded.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\ndegraded: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is degraded\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000150153,\"lastEvaluation\":\"2024-03-27T08:45:28.211520158Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerUnavailable\",\"query\":\"ingress_controller_conditions{condition=\\\"Available\\\"} == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController is not available.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\nunavailable: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is unavailable\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000140454,\"lastEvaluation\":\"2024-03-27T08:45:28.211672746Z\",\"type\":\"alerting\"},{\"name\":\"cluster:openshift_route_info:tls_termination:sum\",\"query\":\"sum by (tls_termination) (openshift_route_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000443213,\"lastEvaluation\":\"2024-03-27T08:45:28.212982901Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:avg\",\"query\":\"avg(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000219295,\"lastEvaluation\":\"2024-03-27T08:45:28.212446152Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:max\",\"query\":\"max(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0002276,\"lastEvaluation\":\"2024-03-27T08:45:28.212214611Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:median\",\"query\":\"quantile(0.5, route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000309002,\"lastEvaluation\":\"2024-03-27T08:45:28.212669552Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:min\",\"query\":\"min(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000341932,\"lastEvaluation\":\"2024-03-27T08:45:28.211868097Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001398715,\"lastEvaluation\":\"2024-03-27T08:45:28.211119359Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"insights\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-60fb2d31-1e44-4f26-84b3-4efa419823b6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"InsightsDisabled\",\"query\":\"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\\\"Disabled\\\",name=\\\"insights\\\"} == 1)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html\",\"summary\":\"Insights operator is disabled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000234656,\"lastEvaluation\":\"2024-03-27T08:45:23.719169132Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"InsightsRecommendationActive\",\"query\":\"insights_recommendation_active == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights recommendation \\\"{{ $labels.description }}\\\" with total risk \\\"{{ $labels.total_risk }}\\\" was detected on the cluster. More information is available at {{ $labels.info_link }}.\",\"summary\":\"An Insights recommendation is active for this cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000269124,\"lastEvaluation\":\"2024-03-27T08:45:23.71977041Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SimpleContentAccessNotAvailable\",\"query\":\"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\\\"SCAAvailable\\\",name=\\\"insights\\\",reason=\\\"NotFound\\\"}[5m]) == 0)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.\",\"summary\":\"Simple content access certificates are not available.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000228197,\"lastEvaluation\":\"2024-03-27T08:45:23.719406563Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000606696,\"lastEvaluation\":\"2024-03-27T08:45:23.719165571Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pre-release-lifecycle\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-f186c8e0-8284-4d6b-bd40-c3ac7b29650d.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"APIRemovedInNextEUSReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=~\\\"1[.]2[89]\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next EUS version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.221433684,\"lastEvaluation\":\"2024-03-27T08:45:34.754812396Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"APIRemovedInNextReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.28\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.177294582,\"lastEvaluation\":\"2024-03-27T08:45:34.577512844Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.209779822,\"lastEvaluation\":\"2024-03-27T08:45:34.577238487Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-audit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-37088249-f88e-4d34-b1f6-f14cf422071c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AuditLogError\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\\\".+-apiserver\\\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\\\".+-apiserver\\\"}[5m])) \\u003e 0\",\"duration\":60,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"An API Server had an error writing to an audit log.\",\"summary\":\"An API Server instance was unable to write audit logs. This could be\\ntriggered by the node running out of space, or a malicious actor\\ntampering with the audit logs.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000657328,\"lastEvaluation\":\"2024-03-27T08:45:31.032664265Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000671473,\"lastEvaluation\":\"2024-03-27T08:45:31.032652937Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"control-plane-cpu-utilization\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-91408eb6-f37d-442b-adbd-0aaeaaa4467f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001312669,\"lastEvaluation\":\"2024-03-27T08:45:42.787819099Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001793645,\"lastEvaluation\":\"2024-03-27T08:45:42.786022253Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneCPU\",\"query\":\"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / count(kube_node_role{role=\\\"master\\\"}) \\u003e 60\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Given three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization across all three control plane nodes is higher than two control plane nodes can sustain; a single control plane node outage may cause a cascading failure; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00154627,\"lastEvaluation\":\"2024-03-27T08:45:42.78429622Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.005290789,\"lastEvaluation\":\"2024-03-27T08:45:42.783844897Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"api-performance\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-recording-rules-719dd770-23f2-4637-8db2-a156da457e7d.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000515674,\"lastEvaluation\":\"2024-03-27T08:45:29.508686061Z\",\"type\":\"recording\"},{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000460971,\"lastEvaluation\":\"2024-03-27T08:45:29.509205218Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.086840182,\"lastEvaluation\":\"2024-03-27T08:45:28.676595832Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.097520636,\"lastEvaluation\":\"2024-03-27T08:45:28.763445182Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:1m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000460828,\"lastEvaluation\":\"2024-03-27T08:45:28.675675607Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:5m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000452751,\"lastEvaluation\":\"2024-03-27T08:45:28.676139594Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011006229,\"lastEvaluation\":\"2024-03-27T08:45:29.419004375Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011773447,\"lastEvaluation\":\"2024-03-27T08:45:29.430014807Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001664526,\"lastEvaluation\":\"2024-03-27T08:45:29.380110341Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001391514,\"lastEvaluation\":\"2024-03-27T08:45:29.383191714Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010421452,\"lastEvaluation\":\"2024-03-27T08:45:29.360215207Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009465122,\"lastEvaluation\":\"2024-03-27T08:45:29.37064083Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.019266848,\"lastEvaluation\":\"2024-03-27T08:45:29.232790735Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.021491489,\"lastEvaluation\":\"2024-03-27T08:45:29.252061712Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.042885479,\"lastEvaluation\":\"2024-03-27T08:45:29.274109399Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.043201669,\"lastEvaluation\":\"2024-03-27T08:45:29.317001778Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000304714,\"lastEvaluation\":\"2024-03-27T08:45:29.273558553Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000240156,\"lastEvaluation\":\"2024-03-27T08:45:29.273866195Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011574124,\"lastEvaluation\":\"2024-03-27T08:45:29.208941347Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.01226562,\"lastEvaluation\":\"2024-03-27T08:45:29.220520322Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012266606,\"lastEvaluation\":\"2024-03-27T08:45:29.468538518Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013907456,\"lastEvaluation\":\"2024-03-27T08:45:29.480809341Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.014539907,\"lastEvaluation\":\"2024-03-27T08:45:29.441792408Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012196271,\"lastEvaluation\":\"2024-03-27T08:45:29.45633771Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013959141,\"lastEvaluation\":\"2024-03-27T08:45:29.494722275Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000470752,\"lastEvaluation\":\"2024-03-27T08:45:28.674768603Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000430054,\"lastEvaluation\":\"2024-03-27T08:45:28.675242405Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012682469,\"lastEvaluation\":\"2024-03-27T08:45:29.058956682Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.089073325,\"lastEvaluation\":\"2024-03-27T08:45:28.860970601Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.094482662,\"lastEvaluation\":\"2024-03-27T08:45:28.950056617Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.031845253,\"lastEvaluation\":\"2024-03-27T08:45:29.072430698Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.032882006,\"lastEvaluation\":\"2024-03-27T08:45:29.104284616Z\",\"type\":\"recording\"},{\"name\":\"instance_request_kind:apiserver_current_inflight_requests:sum\",\"query\":\"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000784225,\"lastEvaluation\":\"2024-03-27T08:45:29.07164338Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.412379512,\"lastEvaluation\":\"2024-03-27T08:45:50.872322001Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.268458584,\"lastEvaluation\":\"2024-03-27T08:45:50.797637129Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":1.29954535,\"lastEvaluation\":\"2024-03-27T08:45:24.698647815Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":2.260180606,\"lastEvaluation\":\"2024-03-27T08:45:25.998202809Z\",\"type\":\"recording\"},{\"name\":\"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum\",\"query\":\"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001409904,\"lastEvaluation\":\"2024-03-27T08:45:29.381778414Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.038146048,\"lastEvaluation\":\"2024-03-27T08:45:28.527352201Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.039127423,\"lastEvaluation\":\"2024-03-27T08:45:28.565504952Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.014403873,\"lastEvaluation\":\"2024-03-27T08:45:29.044547569Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_storage_objects:max\",\"query\":\"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.017342411,\"lastEvaluation\":\"2024-03-27T08:45:29.509669434Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001288621,\"lastEvaluation\":\"2024-03-27T08:45:29.416796445Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000911925,\"lastEvaluation\":\"2024-03-27T08:45:29.418088804Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":1.018052258,\"lastEvaluation\":\"2024-03-27T08:45:48.901774798Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.600876355,\"lastEvaluation\":\"2024-03-27T08:45:49.919838225Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.132773105,\"lastEvaluation\":\"2024-03-27T08:45:28.258391995Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.136162991,\"lastEvaluation\":\"2024-03-27T08:45:28.391175417Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.035456861,\"lastEvaluation\":\"2024-03-27T08:45:29.137171956Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.036296998,\"lastEvaluation\":\"2024-03-27T08:45:29.172637536Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013235596,\"lastEvaluation\":\"2024-03-27T08:45:29.384586617Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.018955345,\"lastEvaluation\":\"2024-03-27T08:45:29.397833671Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":1.245159297,\"lastEvaluation\":\"2024-03-27T08:45:22.492643959Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.960829015,\"lastEvaluation\":\"2024-03-27T08:45:23.737814321Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.235841481,\"lastEvaluation\":\"2024-03-27T08:45:51.06610582Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.534468389,\"lastEvaluation\":\"2024-03-27T08:45:21.958163204Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.034557383,\"lastEvaluation\":\"2024-03-27T08:45:28.604639474Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.035558691,\"lastEvaluation\":\"2024-03-27T08:45:28.639203459Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":5.368955556,\"lastEvaluation\":\"2024-03-27T08:45:18.902060391Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-requests-in-flight\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-ba93f40a-e463-41d7-8a6a-e86ef668197d.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m\",\"query\":\"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver\\\"})[2m:])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001209815,\"lastEvaluation\":\"2024-03-27T08:45:45.551063965Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001232959,\"lastEvaluation\":\"2024-03-27T08:45:45.551047264Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-basic\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-7d793965-450f-4177-83de-ccbd862f1b2a.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1h) \\u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \\u003e (14.4 * 0.01)\",\"duration\":120,\"labels\":{\"long\":\"1h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"5m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000760572,\"lastEvaluation\":\"2024-03-27T08:45:33.637238854Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate6h) \\u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \\u003e (6 * 0.01)\",\"duration\":900,\"labels\":{\"long\":\"6h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"30m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000512383,\"lastEvaluation\":\"2024-03-27T08:45:33.638003604Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001288486,\"lastEvaluation\":\"2024-03-27T08:45:33.637231016Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-7d793965-450f-4177-83de-ccbd862f1b2a.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.280100992,\"lastEvaluation\":\"2024-03-27T08:45:26.421127785Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.500994421,\"lastEvaluation\":\"2024-03-27T08:45:27.206331523Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.148865462,\"lastEvaluation\":\"2024-03-27T08:45:26.27225396Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.343482165,\"lastEvaluation\":\"2024-03-27T08:45:27.707345404Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.104058175,\"lastEvaluation\":\"2024-03-27T08:45:26.168188559Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.104166812,\"lastEvaluation\":\"2024-03-27T08:45:28.050841587Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.505070382,\"lastEvaluation\":\"2024-03-27T08:45:26.701243396Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.223850671,\"lastEvaluation\":\"2024-03-27T08:45:28.155020473Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.512245964,\"lastEvaluation\":\"2024-03-27T08:45:30.267161329Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.551686094,\"lastEvaluation\":\"2024-03-27T08:45:29.715470426Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.539117063,\"lastEvaluation\":\"2024-03-27T08:45:29.176343408Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.391407082,\"lastEvaluation\":\"2024-03-27T08:45:28.466736194Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.318189225,\"lastEvaluation\":\"2024-03-27T08:45:28.858149387Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.046052197,\"lastEvaluation\":\"2024-03-27T08:45:28.378883423Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.041784242,\"lastEvaluation\":\"2024-03-27T08:45:28.424943577Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":4.611240772,\"lastEvaluation\":\"2024-03-27T08:45:26.168177934Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-extended\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-4d9757c2-635f-4e7b-8a9a-1d610b54092b.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1d) \\u003e (3 * 0.01) and sum(apiserver_request:burnrate2h) \\u003e (3 * 0.01)\",\"duration\":3600,\"labels\":{\"long\":\"1d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"2h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000650931,\"lastEvaluation\":\"2024-03-27T08:45:47.233474951Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate3d) \\u003e (1 * 0.01) and sum(apiserver_request:burnrate6h) \\u003e (1 * 0.01)\",\"duration\":10800,\"labels\":{\"long\":\"3d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"6h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000376846,\"lastEvaluation\":\"2024-03-27T08:45:47.234129022Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001047358,\"lastEvaluation\":\"2024-03-27T08:45:47.233462406Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-4d9757c2-635f-4e7b-8a9a-1d610b54092b.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.266518882,\"lastEvaluation\":\"2024-03-27T08:45:22.396745159Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.242420764,\"lastEvaluation\":\"2024-03-27T08:45:22.973692173Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.176789579,\"lastEvaluation\":\"2024-03-27T08:45:22.083445691Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.275907117,\"lastEvaluation\":\"2024-03-27T08:45:23.216123724Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.310400735,\"lastEvaluation\":\"2024-03-27T08:45:22.66327975Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.288269608,\"lastEvaluation\":\"2024-03-27T08:45:23.492042468Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.765764326,\"lastEvaluation\":\"2024-03-27T08:45:22.083430787Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-f87d942a-c815-4a99-b93b-7f0a0b4292bb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TechPreviewNoUpgrade\",\"query\":\"cluster_feature_set{name!=\\\"\\\",namespace=\\\"openshift-kube-apiserver-operator\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.\",\"summary\":\"Cluster has enabled tech preview features that will prevent upgrades.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006021682,\"lastEvaluation\":\"2024-03-27T08:45:25.091445268Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002873574,\"lastEvaluation\":\"2024-03-27T08:45:25.091178929Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pod-security-violation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-f71e3e7e-2d2a-4ae6-96a7-b23c385f9bc8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace!=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created in namespace \\\"{{ $labels.ocp_namespace }}\\\" but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00025296,\"lastEvaluation\":\"2024-03-27T08:45:31.457305558Z\",\"type\":\"alerting\"},{\"state\":\"firing\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads users created in the cluster don't match their Pod Security profile\"},\"alerts\":[{\"labels\":{\"alertname\":\"PodSecurityViolation\",\"namespace\":\"openshift-kube-apiserver\",\"policy_level\":\"restricted\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \\\"restricted\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads users created in the cluster don't match their Pod Security profile\"},\"state\":\"firing\",\"activeAt\":\"2024-03-27T08:26:31.452115983Z\",\"value\":\"1.0347239285714286e+01\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.001061565,\"lastEvaluation\":\"2024-03-27T08:45:31.45624014Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001335471,\"lastEvaluation\":\"2024-03-27T08:45:31.456227129Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-6a4bea6d-69ac-4b07-8ab7-34e717b065d0.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"GarbageCollectorSyncFailed\",\"query\":\"rate(garbagecollector_controller_resources_sync_error_total[5m]) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md\",\"summary\":\"There was a problem with syncing the resources for garbage collection.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000396859,\"lastEvaluation\":\"2024-03-27T08:45:50.80447189Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeControllerManagerDown\",\"query\":\"absent(up{job=\\\"kube-controller-manager\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-controller-manager\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeControllerManager has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000270293,\"lastEvaluation\":\"2024-03-27T08:45:50.799622775Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetAtLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \\u003e 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md\",\"summary\":\"The pod disruption budget is preventing further disruption to pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002433948,\"lastEvaluation\":\"2024-03-27T08:45:50.800191195Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \\u003c kube_poddisruptionbudget_status_desired_healthy)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md\",\"summary\":\"The pod disruption budget registers insufficient amount of pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001832415,\"lastEvaluation\":\"2024-03-27T08:45:50.80263226Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002342314,\"lastEvaluation\":\"2024-03-27T08:45:50.799618506Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-cd5bc467-4fd9-4006-9cdb-67ca18b3a004.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeSchedulerDown\",\"query\":\"absent(up{job=\\\"scheduler\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-scheduler\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeScheduler has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000215893,\"lastEvaluation\":\"2024-03-27T08:45:43.160710229Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000463713,\"lastEvaluation\":\"2024-03-27T08:45:43.160009979Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"scheduler-legacy-policy-deprecated\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-cd5bc467-4fd9-4006-9cdb-67ca18b3a004.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SchedulerLegacyPolicySet\",\"query\":\"cluster_legacy_scheduler_policy \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.\",\"summary\":\"Legacy scheduler policy API in use by the scheduler.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000243046,\"lastEvaluation\":\"2024-03-27T08:45:32.913103609Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000657758,\"lastEvaluation\":\"2024-03-27T08:45:32.912920123Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-api-operator-metrics-collector-up\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineAPIOperatorMetricsCollectionFailing\",\"query\":\"mapi_mao_collector_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"For more details:  oc logs \\u003cmachine-api-operator-pod-name\\u003e -n openshift-machine-api\",\"summary\":\"machine api operator metrics collection is failing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000859831,\"lastEvaluation\":\"2024-03-27T08:45:27.3535061Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000364509,\"lastEvaluation\":\"2024-03-27T08:45:27.347706727Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-health-check-unterminated-short-circuit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineHealthCheckUnterminatedShortCircuit\",\"query\":\"mapi_machinehealthcheck_short_circuit == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\\nthe status of machines in the cluster.\\n\",\"summary\":\"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000362663,\"lastEvaluation\":\"2024-03-27T08:45:32.708101452Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000406098,\"lastEvaluation\":\"2024-03-27T08:45:32.708070201Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-not-yet-deleted\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineNotYetDeleted\",\"query\":\"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\\\"Deleting\\\"}[15m])) \\u003e 0\",\"duration\":21600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine is not properly deleting, this may be due to a configuration issue with the\\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\\nlong termination periods which are preventing deletion.\\n\",\"summary\":\"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000347677,\"lastEvaluation\":\"2024-03-27T08:45:39.596140743Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000564677,\"lastEvaluation\":\"2024-03-27T08:45:39.596103747Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-with-no-running-phase\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithNoRunningPhase\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\\\"Running|Deleting\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine has been without a Running or Deleting phase for more than 60 minutes.\\nThe machine may not have been provisioned properly from the infrastructure provider, or\\nit might have issues with CertificateSigningRequests being approved.\\n\",\"summary\":\"machine {{ $labels.name }} is in phase: {{ $labels.phase }}\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000222833,\"lastEvaluation\":\"2024-03-27T08:45:24.892835516Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000785693,\"lastEvaluation\":\"2024-03-27T08:45:24.892718281Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-without-valid-node-ref\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-6cb2691e-1be6-423b-92ed-e7ec06c41f42.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithoutValidNode\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"If the machine never became a node, you should diagnose the machine related failures.\\nIf the node was deleted from the API, you may delete the machine if appropriate.\\n\",\"summary\":\"machine {{ $labels.name }} does not have valid node reference\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000791042,\"lastEvaluation\":\"2024-03-27T08:45:28.445891363Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000810741,\"lastEvaluation\":\"2024-03-27T08:45:28.445880983Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-drain-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-c3ddad37-88e0-4b4a-a21a-ef5f971392f8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCDrainError\",\"query\":\"mcc_drain_err \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000324526,\"lastEvaluation\":\"2024-03-27T08:45:36.421585265Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000231023,\"lastEvaluation\":\"2024-03-27T08:45:36.421472523Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-pool-alert\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-c3ddad37-88e0-4b4a-a21a-ef5f971392f8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCPoolAlert\",\"query\":\"mcc_pool_alert \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000577471,\"lastEvaluation\":\"2024-03-27T08:45:24.443110855Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000545323,\"lastEvaluation\":\"2024-03-27T08:45:24.44264836Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"os-image-override.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-c3ddad37-88e0-4b4a-a21a-ef5f971392f8.yaml\",\"rules\":[{\"name\":\"os_image_url_override:sum\",\"query\":\"sum(os_image_url_override)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000607114,\"lastEvaluation\":\"2024-03-27T08:45:42.659325222Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000431801,\"lastEvaluation\":\"2024-03-27T08:45:42.658780681Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"extremely-high-individual-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneMemory\",\"query\":\"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 90\",\"duration\":2700,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.\",\"summary\":\"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00222741,\"lastEvaluation\":\"2024-03-27T08:45:26.130717627Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001381396,\"lastEvaluation\":\"2024-03-27T08:45:26.129816223Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"high-overall-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneMemory\",\"query\":\"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 60\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.\",\"summary\":\"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00108057,\"lastEvaluation\":\"2024-03-27T08:45:39.842642699Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001095726,\"lastEvaluation\":\"2024-03-27T08:45:39.842630014Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-kubelet-health-state-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeletHealthState\",\"query\":\"mcd_kubelet_state \\u003e 2\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet health failure threshold reached\",\"summary\":\"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000606619,\"lastEvaluation\":\"2024-03-27T08:45:33.974737203Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000629891,\"lastEvaluation\":\"2024-03-27T08:45:33.974719954Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-pivot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDPivotError\",\"query\":\"mcd_pivot_errors_total \\u003e 0\",\"duration\":120,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000276504,\"lastEvaluation\":\"2024-03-27T08:45:31.24104656Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000286744,\"lastEvaluation\":\"2024-03-27T08:45:31.241039088Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-reboot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDRebootError\",\"query\":\"mcd_reboots_failed_total \\u003e 0\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000289734,\"lastEvaluation\":\"2024-03-27T08:45:44.852829304Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000302216,\"lastEvaluation\":\"2024-03-27T08:45:44.852819749Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"system-memory-exceeds-reservation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-c1dc03e8-7a7a-4983-aa83-8a3081aeaed6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SystemMemoryExceedsReservation\",\"query\":\"sum by (node) (container_memory_rss{id=\\\"/system.slice\\\"}) \\u003e ((sum by (node) (kube_node_status_capacity{resource=\\\"memory\\\"} - kube_node_status_allocatable{resource=\\\"memory\\\"})) * 0.95)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).\",\"summary\":\"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000941558,\"lastEvaluation\":\"2024-03-27T08:45:31.527339494Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00095989,\"lastEvaluation\":\"2024-03-27T08:45:31.527325789Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"operator.marketplace.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-marketplace-marketplace-alert-rules-5f4ea3a5-aee7-484b-96bc-522d41b231cc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"OperatorHubSourceError\",\"query\":\"catalogsource_ready{exported_namespace=\\\"openshift-marketplace\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Operators shipped via the {{ $labels.name }} source are not available for installation until the issue is fixed. Operators already installed from this source will not receive updates until issue is fixed. Inspect the status of the pod owned by {{ $labels.name }} source in the openshift-marketplace namespace (oc -n openshift-marketplace get pods -l olm.catalogSource={{ $labels.name }}) to diagnose and repair.\",\"summary\":\"The {{ $labels.name }} source is in non-ready state for more than 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000743302,\"lastEvaluation\":\"2024-03-27T08:45:42.595064825Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000405628,\"lastEvaluation\":\"2024-03-27T08:45:42.594435753Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"alertmanager.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-alertmanager-main-rules-0472e654-98ee-4d67-8369-8aebd739a6d2.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterDown\",\"query\":\"(count by (namespace, service) (avg_over_time(up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c 0.5) / count by (namespace, service) (up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) \\u003e= 0.5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of Alertmanager instances within the {{$labels.job}} cluster have been up for less than half of the last 5m.\",\"summary\":\"Half or more of the Alertmanager instances within the same cluster are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000890144,\"lastEvaluation\":\"2024-03-27T08:45:29.251030565Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterFailedToSendAlerts\",\"query\":\"min by (namespace, service, integration) (rate(alertmanager_notifications_failed_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / rate(alertmanager_notifications_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The minimum notification failure rate to {{ $labels.integration }} sent from any instance in the {{$labels.job}} cluster is {{ $value | humanizePercentage }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerClusterFailedToSendAlerts.md\",\"summary\":\"All Alertmanager instances in a cluster failed to send notifications to a critical integration.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001968288,\"lastEvaluation\":\"2024-03-27T08:45:29.248564779Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerConfigInconsistent\",\"query\":\"count by (namespace, service) (count_values by (namespace, service) (\\\"config_hash\\\", alertmanager_config_hash{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) != 1\",\"duration\":1200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager instances within the {{$labels.job}} cluster have different configurations.\",\"summary\":\"Alertmanager instances within the same cluster have different configurations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000489018,\"lastEvaluation\":\"2024-03-27T08:45:29.250537514Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedReload\",\"query\":\"max_over_time(alertmanager_config_last_reload_successful{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Configuration has failed to load for {{ $labels.namespace }}/{{ $labels.pod}}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedReload.md\",\"summary\":\"Reloading an Alertmanager configuration has failed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000344675,\"lastEvaluation\":\"2024-03-27T08:45:29.245727162Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedToSendAlerts\",\"query\":\"(rate(alertmanager_notifications_failed_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / rate(alertmanager_notifications_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} failed to send {{ $value | humanizePercentage }} of notifications to {{ $labels.integration }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedToSendAlerts.md\",\"summary\":\"An Alertmanager instance failed to send notifications.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001872041,\"lastEvaluation\":\"2024-03-27T08:45:29.246688486Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerMembersInconsistent\",\"query\":\"max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c on (namespace, service) group_left () count by (namespace, service) (max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]))\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} has only found {{ $value }} members of the {{$labels.job}} cluster.\",\"summary\":\"A member of an Alertmanager cluster has not found all other cluster members.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000347718,\"lastEvaluation\":\"2024-03-27T08:45:29.246074025Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002991198,\"lastEvaluation\":\"2024-03-27T08:45:29.245718092Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-list-watch.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"apiserver_list_watch_request_success_total:rate:sum\",\"query\":\"sum by (verb) (rate(apiserver_request_total{code=~\\\"2..\\\",verb=~\\\"LIST|WATCH\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.026909,\"lastEvaluation\":\"2024-03-27T08:45:45.807634848Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.04322445,\"lastEvaluation\":\"2024-03-27T08:45:45.806812425Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"Watchdog\",\"query\":\"vector(1)\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"alerts\":[{\"labels\":{\"alertname\":\"Watchdog\",\"namespace\":\"openshift-monitoring\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"state\":\"firing\",\"activeAt\":\"2024-03-27T07:22:21.164916952Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000758705,\"lastEvaluation\":\"2024-03-27T08:45:51.166943531Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000779716,\"lastEvaluation\":\"2024-03-27T08:45:51.166928436Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"count:up0\",\"query\":\"count without (instance, pod, node) (up == 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002950463,\"lastEvaluation\":\"2024-03-27T08:45:32.981802677Z\",\"type\":\"recording\"},{\"name\":\"count:up1\",\"query\":\"count without (instance, pod, node) (up == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003070788,\"lastEvaluation\":\"2024-03-27T08:45:32.978286212Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006569208,\"lastEvaluation\":\"2024-03-27T08:45:32.97818814Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-node-recording.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"cluster:node_cpu:ratio\",\"query\":\"cluster:node_cpu:sum_rate5m / count(sum by (instance, cpu) (node_cpu_seconds_total))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009158959,\"lastEvaluation\":\"2024-03-27T08:45:28.874578197Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:sum_rate5m\",\"query\":\"sum(rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006759567,\"lastEvaluation\":\"2024-03-27T08:45:28.867811301Z\",\"type\":\"recording\"},{\"name\":\"instance:node_cpu:rate:sum\",\"query\":\"sum by (instance) (rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002691087,\"lastEvaluation\":\"2024-03-27T08:45:28.859294055Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_receive_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001294304,\"lastEvaluation\":\"2024-03-27T08:45:28.865235684Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_transmit_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_transmit_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001272228,\"lastEvaluation\":\"2024-03-27T08:45:28.866534869Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.025226908,\"lastEvaluation\":\"2024-03-27T08:45:28.858518251Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-recurring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"cluster:usage:workload:capacity_physical_cpu_core_seconds\",\"query\":\"sum_over_time(workload:capacity_physical_cpu_cores:sum[30s:1s]) + ((cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) or (absent(cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) * 0))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001044034,\"lastEvaluation\":\"2024-03-27T08:45:41.448431901Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001067658,\"lastEvaluation\":\"2024-03-27T08:45:41.448415923Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-network\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeNetworkInterfaceFlapping\",\"query\":\"changes(node_network_up{device!~\\\"veth.+|tunbr\\\",job=\\\"node-exporter\\\"}[2m]) \\u003e 2\",\"duration\":120,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Network interface \\\"{{ $labels.device }}\\\" changing its up status often on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\",\"summary\":\"Network interface is often changing its status\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001129163,\"lastEvaluation\":\"2024-03-27T08:45:30.225782206Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001138163,\"lastEvaluation\":\"2024-03-27T08:45:30.225777416Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-build.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"openshift:build_by_strategy:sum\",\"query\":\"sum by (strategy) (openshift_build_status_phase_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001043621,\"lastEvaluation\":\"2024-03-27T08:45:49.293580313Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000603851,\"lastEvaluation\":\"2024-03-27T08:45:49.28737229Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd-telemetry.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001371138,\"lastEvaluation\":\"2024-03-27T08:45:31.534270809Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001900661,\"lastEvaluation\":\"2024-03-27T08:45:31.529373116Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000384632,\"lastEvaluation\":\"2024-03-27T08:45:31.528983793Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_use_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00026768,\"lastEvaluation\":\"2024-03-27T08:45:31.533999966Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.002717597,\"lastEvaluation\":\"2024-03-27T08:45:31.531278925Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006666372,\"lastEvaluation\":\"2024-03-27T08:45:31.528979025Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TargetDown\",\"query\":\"100 * ((1 - sum by (job, namespace, service) (up and on (namespace, pod) kube_pod_info) / count by (job, namespace, service) (up and on (namespace, pod) kube_pod_info)) or (count by (job, namespace, service) (up == 0) / count by (job, namespace, service) (up))) \\u003e 10\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ printf \\\"%.4g\\\" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.\",\"summary\":\"Some targets were not reachable from the monitoring server for an extended period of time.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.034926089,\"lastEvaluation\":\"2024-03-27T08:45:38.637381409Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.020686266,\"lastEvaluation\":\"2024-03-27T08:45:38.637345854Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"cluster:ingress_controller_aws_nlb_active:sum\",\"query\":\"sum(ingress_controller_aws_nlb_active) or vector(0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000607386,\"lastEvaluation\":\"2024-03-27T08:45:24.898300637Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_in:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_in_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000885292,\"lastEvaluation\":\"2024-03-27T08:45:24.859711714Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_out:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_out_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000832808,\"lastEvaluation\":\"2024-03-27T08:45:24.860602908Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_connections:sum\",\"query\":\"sum(haproxy_frontend_current_sessions)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000622141,\"lastEvaluation\":\"2024-03-27T08:45:24.861444328Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace=~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.015124351,\"lastEvaluation\":\"2024-03-27T08:45:24.870711839Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.01244188,\"lastEvaluation\":\"2024-03-27T08:45:24.885846178Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace!~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005143992,\"lastEvaluation\":\"2024-03-27T08:45:24.862073765Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003480493,\"lastEvaluation\":\"2024-03-27T08:45:24.867223962Z\",\"type\":\"recording\"},{\"name\":\"code:cluster:ingress_http_request_count:rate5m:sum\",\"query\":\"sum by (code) (rate(haproxy_server_http_responses_total[5m]) \\u003e 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010005955,\"lastEvaluation\":\"2024-03-27T08:45:24.849696854Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.049231665,\"lastEvaluation\":\"2024-03-27T08:45:24.849683427Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-kubernetes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"AlertmanagerReceiversNotConfigured\",\"query\":\"cluster:alertmanager_integrations:max == 0\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"alerts\":[{\"labels\":{\"alertname\":\"AlertmanagerReceiversNotConfigured\",\"namespace\":\"openshift-monitoring\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"state\":\"firing\",\"activeAt\":\"2024-03-27T07:22:38.472010051Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000469268,\"lastEvaluation\":\"2024-03-27T08:45:38.909460701Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterMonitoringOperatorReconciliationErrors\",\"query\":\"max_over_time(cluster_monitoring_operator_last_reconciliation_successful[5m]) == 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors are occurring during reconciliation cycles. Inspect the cluster-monitoring-operator log for potential root causes.\",\"summary\":\"Cluster Monitoring Operator is experiencing unexpected reconciliation errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000223699,\"lastEvaluation\":\"2024-03-27T08:45:38.909234087Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentReplicasMismatch\",\"query\":\"(((kube_deployment_spec_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_deployment_status_replicas_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_deployment_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)) * on () group_left () cluster:control_plane:all_nodes_ready) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes. This indicates that cluster infrastructure is unable to start or restart the necessary components. This most often occurs when one or more nodes are down or partioned from the cluster, or a fault occurs on the node that prevents the workload from starting. In rare cases this may indicate a new version of a cluster component cannot start due to a bug or configuration error. Assess the pods for this deployment to verify they are running on healthy nodes and then contact support.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeDeploymentReplicasMismatch.md\",\"summary\":\"Deployment has not matched the expected number of replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00864537,\"lastEvaluation\":\"2024-03-27T08:45:38.909933175Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodNotScheduled\",\"query\":\"last_over_time(kube_pod_status_unschedulable{namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} cannot be scheduled for more than 30 minutes.\\nCheck the details of the pod with the following command:\\noc describe -n {{ $labels.namespace }} pod {{ $labels.pod }}\",\"summary\":\"Pod cannot be scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00122576,\"lastEvaluation\":\"2024-03-27T08:45:39.018697044Z\",\"type\":\"alerting\"},{\"name\":\"cluster:alertmanager_integrations:max\",\"query\":\"max(alertmanager_integrations{namespace=\\\"openshift-monitoring\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00020203,\"lastEvaluation\":\"2024-03-27T08:45:38.907969987Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (label_replace(cluster:infra_nodes, \\\"label_node_role_kubernetes_io\\\", \\\"infra\\\", \\\"\\\", \\\"\\\") * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001324817,\"lastEvaluation\":\"2024-03-27T08:45:38.77677198Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000787115,\"lastEvaluation\":\"2024-03-27T08:45:38.783383305Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_sockets_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (max by (node, package, label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00089522,\"lastEvaluation\":\"2024-03-27T08:45:38.906161087Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_memory_bytes:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000918405,\"lastEvaluation\":\"2024-03-27T08:45:38.784173747Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_cpu_usage:ratio\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m])) / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.042044936,\"lastEvaluation\":\"2024-03-27T08:45:38.730962937Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_spec_cpu_shares:ratio\",\"query\":\"sum(container_spec_cpu_shares{container=\\\"\\\",pod!=\\\"\\\"}) / 1000 / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.044153161,\"lastEvaluation\":\"2024-03-27T08:45:38.686805032Z\",\"type\":\"recording\"},{\"name\":\"cluster:control_plane:all_nodes_ready\",\"query\":\"sum(min by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}) and max by (node) (kube_node_role{role=\\\"master\\\"})) == bool sum(kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000558666,\"lastEvaluation\":\"2024-03-27T08:45:38.908482872Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_hyperthreading\",\"query\":\"clamp_max(label_replace(sum by (instance, package, core) (node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_replace(label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003e 1, \\\"label_node_hyperthread_enabled\\\", \\\"true\\\", \\\"instance\\\", \\\"(.*)\\\") or on (instance, package) label_replace(sum by (instance, package, core) (label_replace(node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003c= 1, \\\"label_node_hyperthread_enabled\\\", \\\"false\\\", \\\"instance\\\", \\\"(.*)\\\"), 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003599898,\"lastEvaluation\":\"2024-03-27T08:45:38.778100353Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_node_labels\",\"query\":\"topk by (node) (1, cluster:nodes_roles) * on (node) group_right (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_node_openshift_io_os_id, label_kubernetes_io_arch, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) label_replace(cluster:cpu_core_hyperthreading, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00167668,\"lastEvaluation\":\"2024-03-27T08:45:38.781703512Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_usage_cores:sum\",\"query\":\"sum(1 - rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[2m]) * on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{pod=~\\\"node-exporter.+\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001906469,\"lastEvaluation\":\"2024-03-27T08:45:38.785095177Z\",\"type\":\"recording\"},{\"name\":\"cluster:hyperthread_enabled_nodes\",\"query\":\"kube_node_labels and on (node) (sum by (node, package, core) (label_replace(node_cpu_info, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")) == 2)\",\"labels\":{\"label_node_hyperthread_enabled\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001765164,\"lastEvaluation\":\"2024-03-27T08:45:38.774603832Z\",\"type\":\"recording\"},{\"name\":\"cluster:infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"infra\\\"})\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00030626,\"lastEvaluation\":\"2024-03-27T08:45:38.773462579Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolume_plugin_type_counts:sum\",\"query\":\"sum by (plugin_name, volume_mode) (pv_collector_total_pv_count)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000304869,\"lastEvaluation\":\"2024-03-27T08:45:38.90817509Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_resource_requests_storage_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001485419,\"lastEvaluation\":\"2024-03-27T08:45:38.875286571Z\",\"type\":\"recording\"},{\"name\":\"cluster:kubelet_volume_stats_used_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kubelet_volume_stats_used_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001144583,\"lastEvaluation\":\"2024-03-27T08:45:38.878217459Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (cluster:master_nodes and on (node) cluster:infra_nodes)\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000265586,\"lastEvaluation\":\"2024-03-27T08:45:38.773771756Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"label_node_role_kubernetes_io\":\"master\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000446442,\"lastEvaluation\":\"2024-03-27T08:45:38.773012938Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage:ratio\",\"query\":\"sum by (cluster) (container_memory_usage_bytes{container=\\\"\\\",pod!=\\\"\\\"}) / sum by (cluster) (machine_memory_bytes)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.046905606,\"lastEvaluation\":\"2024-03-27T08:45:38.63989404Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage_bytes:sum\",\"query\":\"sum(node_memory_MemTotal_bytes{job=\\\"node-exporter\\\"} - node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000452129,\"lastEvaluation\":\"2024-03-27T08:45:38.787004775Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_instance_type_count:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) (cluster:master_nodes or on (node) kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000404111,\"lastEvaluation\":\"2024-03-27T08:45:38.874879372Z\",\"type\":\"recording\"},{\"name\":\"cluster:nodes_roles\",\"query\":\"cluster:master_infra_nodes or on (node) cluster:master_nodes or on (node) cluster:infra_nodes or on (node) max without (endpoint, instance, job, pod, service) (kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000560411,\"lastEvaluation\":\"2024-03-27T08:45:38.774040114Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:containers:sum\",\"query\":\"sum(max by (instance) (kubelet_containers_per_pod_count_sum))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00045936,\"lastEvaluation\":\"2024-03-27T08:45:38.904542567Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_node_ready:avg5m\",\"query\":\"avg_over_time((count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1)) / scalar(count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010728877,\"lastEvaluation\":\"2024-03-27T08:45:38.946202011Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_schedulable_node_ready_reachable:avg5m\",\"query\":\"avg_over_time((((count((max by (node) (up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1) and max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1) and min by (node) (kube_node_spec_unschedulable == 0))) / scalar(count(min by (node) (kube_node_spec_unschedulable == 0))))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.027611442,\"lastEvaluation\":\"2024-03-27T08:45:38.918582096Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace=~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005208091,\"lastEvaluation\":\"2024-03-27T08:45:39.008385243Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:pods:terminal:workload:sum\",\"query\":\"count(count by (namespace, pod) (kube_pod_restart_policy{namespace!~\\\"openshift-.+\\\",type!=\\\"Always\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000768445,\"lastEvaluation\":\"2024-03-27T08:45:38.903768999Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:resources:sum\",\"query\":\"topk(500, max by (resource) (apiserver_storage_objects))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012878969,\"lastEvaluation\":\"2024-03-27T08:45:38.890884116Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:max:5m\",\"query\":\"max_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000190541,\"lastEvaluation\":\"2024-03-27T08:45:38.878023737Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:min:5m\",\"query\":\"min_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000211438,\"lastEvaluation\":\"2024-03-27T08:45:38.877809234Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace!~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005095546,\"lastEvaluation\":\"2024-03-27T08:45:39.013597457Z\",\"type\":\"recording\"},{\"name\":\"cluster:virt_platform_nodes:sum\",\"query\":\"count by (type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (sum by (instance, type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (virt_platform))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000394327,\"lastEvaluation\":\"2024-03-27T08:45:38.776374274Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_object_counts:sum\",\"query\":\"sum by (instance) (apiserver_storage_objects)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011515503,\"lastEvaluation\":\"2024-03-27T08:45:38.879365075Z\",\"type\":\"recording\"},{\"name\":\"kube_running_pod_ready\",\"query\":\"(max without (condition, container, endpoint, instance, job, service) (((kube_pod_status_ready{condition=\\\"false\\\"} == 1) * 0 or (kube_pod_status_ready{condition=\\\"true\\\"} == 1)) * on (pod, namespace) group_left () group by (pod, namespace) (kube_pod_status_phase{phase=~\\\"Running|Unknown|Pending\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.051443961,\"lastEvaluation\":\"2024-03-27T08:45:38.956935245Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_cpu_usage:sum\",\"query\":\"sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=\\\"\\\",container!=\\\"POD\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.026301636,\"lastEvaluation\":\"2024-03-27T08:45:38.613586089Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_memory_usage_bytes:sum\",\"query\":\"sum by (namespace) (container_memory_usage_bytes{container!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.043659328,\"lastEvaluation\":\"2024-03-27T08:45:38.569918488Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_cores:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001150362,\"lastEvaluation\":\"2024-03-27T08:45:38.905006594Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_sockets:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (max by (node, package, label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000907233,\"lastEvaluation\":\"2024-03-27T08:45:38.907059562Z\",\"type\":\"recording\"},{\"name\":\"openshift:cpu_usage_cores:sum\",\"query\":\"cluster:cpu_usage_cores:sum - workload:cpu_usage_cores:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000341643,\"lastEvaluation\":\"2024-03-27T08:45:38.829965851Z\",\"type\":\"recording\"},{\"name\":\"openshift:memory_usage_bytes:sum\",\"query\":\"cluster:memory_usage_bytes:sum - workload:memory_usage_bytes:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000321816,\"lastEvaluation\":\"2024-03-27T08:45:38.87455433Z\",\"type\":\"recording\"},{\"name\":\"pod:container_cpu_usage:sum\",\"query\":\"sum by (pod, namespace) (rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.027299801,\"lastEvaluation\":\"2024-03-27T08:45:38.473614406Z\",\"type\":\"recording\"},{\"name\":\"pod:container_fs_usage_bytes:sum\",\"query\":\"sum by (pod, namespace) (container_fs_usage_bytes{pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.05082194,\"lastEvaluation\":\"2024-03-27T08:45:38.519080827Z\",\"type\":\"recording\"},{\"name\":\"profile:cluster_monitoring_operator_collection_profile:max\",\"query\":\"max by (profile) (cluster_monitoring_operator_collection_profile == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000185952,\"lastEvaluation\":\"2024-03-27T08:45:38.909044859Z\",\"type\":\"recording\"},{\"name\":\"workload:capacity_physical_cpu_cores:sum\",\"query\":\"(sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_infra=\\\"\\\",label_node_role_kubernetes_io_master=\\\"\\\"} or absent(__does_not_exist__) * 0)) + ((sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_master=\\\"true\\\"} or absent(__does_not_exist__) * 0) * ((max(cluster_master_schedulable == 1) * 0 + 1) or (absent(cluster_master_schedulable == 1) * 0))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001030688,\"lastEvaluation\":\"2024-03-27T08:45:38.876775288Z\",\"type\":\"recording\"},{\"name\":\"workload:cpu_usage_cores:sum\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.042499417,\"lastEvaluation\":\"2024-03-27T08:45:38.787460008Z\",\"type\":\"recording\"},{\"name\":\"workload:memory_usage_bytes:sum\",\"query\":\"sum(container_memory_working_set_bytes{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.044238413,\"lastEvaluation\":\"2024-03-27T08:45:38.830310619Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.316973138,\"lastEvaluation\":\"2024-03-27T08:45:38.473601633Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-monitoring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"monitoring:container_memory_working_set_bytes:sum\",\"query\":\"sum by (namespace) (max without (instance) (container_memory_working_set_bytes{container=\\\"\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003619478,\"lastEvaluation\":\"2024-03-27T08:45:31.779843038Z\",\"type\":\"recording\"},{\"name\":\"monitoring:haproxy_server_http_responses_total:sum\",\"query\":\"sum by (exported_service) (rate(haproxy_server_http_responses_total{exported_namespace=\\\"openshift-monitoring\\\",exported_service=~\\\"alertmanager-main|prometheus-k8s\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002337153,\"lastEvaluation\":\"2024-03-27T08:45:31.799866279Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_samples_post_metric_relabeling:topk3\",\"query\":\"topk(3, max by (namespace, job) (topk by (namespace, job) (1, scrape_samples_post_metric_relabeling)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003999327,\"lastEvaluation\":\"2024-03-27T08:45:31.795863284Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_series_added:topk3_sum1h\",\"query\":\"topk(3, sum by (namespace, job) (sum_over_time(scrape_series_added[1h])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012390659,\"lastEvaluation\":\"2024-03-27T08:45:31.783467336Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicationController\\\"}, \\\"replicationcontroller\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicationcontroller, namespace) group_left (owner_name) topk by (replicationcontroller, namespace) (1, max by (replicationcontroller, namespace, owner_name) (kube_replicationcontroller_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deploymentconfig\"},\"health\":\"ok\",\"evaluationTime\":0.002530907,\"lastEvaluation\":\"2024-03-27T08:45:31.802208423Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_samples_appended_total:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (rate(prometheus_tsdb_head_samples_appended_total{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[2m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000398768,\"lastEvaluation\":\"2024-03-27T08:45:31.779437589Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_series:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (prometheus_tsdb_head_series{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000521319,\"lastEvaluation\":\"2024-03-27T08:45:31.778912411Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.012781179,\"lastEvaluation\":\"2024-03-27T08:45:31.778506926Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-sre.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"code:apiserver_request_total:rate:sum\",\"query\":\"sum by (code) (rate(apiserver_request_total{job=\\\"apiserver\\\"}[10m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.086422803,\"lastEvaluation\":\"2024-03-27T08:45:30.036743364Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.049049596,\"lastEvaluation\":\"2024-03-27T08:45:30.036704345Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-vsphere.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-5972f260-b346-4b58-bc4d-23fffe1e4d1f.yaml\",\"rules\":[{\"name\":\"cluster:vsphere_csi_migration:max\",\"query\":\"max by (status) (vsphere_csi_migration{status=~\\\"|LegacyDeprecatedInTreeDriver|CSIWithMigrationDriver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000568996,\"lastEvaluation\":\"2024-03-27T08:45:22.403011826Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_esxi_version_total:sum\",\"query\":\"sum by (version) (vsphere_esxi_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000120959,\"lastEvaluation\":\"2024-03-27T08:45:22.402385945Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_infrastructure_failure_domains:max\",\"query\":\"max by (scope) (vsphere_infrastructure_failure_domains)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000119476,\"lastEvaluation\":\"2024-03-27T08:45:22.402768256Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_node_hw_version_total:sum\",\"query\":\"sum by (hw_version) (vsphere_node_hw_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000128957,\"lastEvaluation\":\"2024-03-27T08:45:22.402510177Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_topology_tags:max\",\"query\":\"max by (source) (vsphere_topology_tags)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000123939,\"lastEvaluation\":\"2024-03-27T08:45:22.402641653Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_vcenter_info:sum\",\"query\":\"sum by (version, build) (vsphere_vcenter_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000246378,\"lastEvaluation\":\"2024-03-27T08:45:22.402136945Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000973216,\"lastEvaluation\":\"2024-03-27T08:45:22.402133125Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-state-metrics\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kube-state-metrics-rules-efd623b1-12df-4ae4-ba23-205c862f8327.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeStateMetricsListErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in list operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in list operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000664315,\"lastEvaluation\":\"2024-03-27T08:45:33.620853113Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStateMetricsWatchErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in watch operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in watch operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000513111,\"lastEvaluation\":\"2024-03-27T08:45:33.621522092Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00251924,\"lastEvaluation\":\"2024-03-27T08:45:33.619519977Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.038487375,\"lastEvaluation\":\"2024-03-27T08:45:23.70772229Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.055410374,\"lastEvaluation\":\"2024-03-27T08:45:23.514685Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.037652177,\"lastEvaluation\":\"2024-03-27T08:45:23.635339597Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.056286061,\"lastEvaluation\":\"2024-03-27T08:45:23.40344913Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.035085797,\"lastEvaluation\":\"2024-03-27T08:45:23.74621606Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.065220889,\"lastEvaluation\":\"2024-03-27T08:45:23.570106597Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.03471478,\"lastEvaluation\":\"2024-03-27T08:45:23.67299929Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.054930196,\"lastEvaluation\":\"2024-03-27T08:45:23.459745233Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"DaemonSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"daemonset\"},\"health\":\"ok\",\"evaluationTime\":0.002960942,\"lastEvaluation\":\"2024-03-27T08:45:23.791195078Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicaSet\\\"}, \\\"replicaset\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicaset, namespace) group_left (owner_name) topk by (replicaset, namespace) (1, max by (replicaset, namespace, owner_name) (kube_replicaset_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deployment\"},\"health\":\"ok\",\"evaluationTime\":0.009879996,\"lastEvaluation\":\"2024-03-27T08:45:23.781310472Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"Job\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"job\"},\"health\":\"ok\",\"evaluationTime\":0.000706558,\"lastEvaluation\":\"2024-03-27T08:45:23.795465836Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"StatefulSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"statefulset\"},\"health\":\"ok\",\"evaluationTime\":0.001294074,\"lastEvaluation\":\"2024-03-27T08:45:23.794166953Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate\",\"query\":\"sum by (cluster, namespace, pod, container) (irate(container_cpu_usage_seconds_total{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"}[5m])) * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.06250654,\"lastEvaluation\":\"2024-03-27T08:45:23.077364252Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_memory_cache\",\"query\":\"container_memory_cache{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.064029051,\"lastEvaluation\":\"2024-03-27T08:45:23.262278578Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_memory_rss\",\"query\":\"container_memory_rss{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.061969574,\"lastEvaluation\":\"2024-03-27T08:45:23.200300598Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_memory_swap\",\"query\":\"container_memory_swap{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.07711624,\"lastEvaluation\":\"2024-03-27T08:45:23.326321211Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod_container:container_memory_working_set_bytes\",\"query\":\"container_memory_working_set_bytes{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.060407376,\"lastEvaluation\":\"2024-03-27T08:45:23.139881172Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.275982231,\"lastEvaluation\":\"2024-03-27T08:45:23.076615453Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-scheduler.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.00030357,\"lastEvaluation\":\"2024-03-27T08:45:30.098262654Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000312908,\"lastEvaluation\":\"2024-03-27T08:45:30.095884435Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.00040136,\"lastEvaluation\":\"2024-03-27T08:45:30.0929236Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000301222,\"lastEvaluation\":\"2024-03-27T08:45:30.096201631Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000380069,\"lastEvaluation\":\"2024-03-27T08:45:30.093329985Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000242133,\"lastEvaluation\":\"2024-03-27T08:45:30.090885143Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.001751755,\"lastEvaluation\":\"2024-03-27T08:45:30.096506599Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.00216515,\"lastEvaluation\":\"2024-03-27T08:45:30.093715009Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000825024,\"lastEvaluation\":\"2024-03-27T08:45:30.091129252Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.008549868,\"lastEvaluation\":\"2024-03-27T08:45:30.090021401Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubelet.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.004432477,\"lastEvaluation\":\"2024-03-27T08:45:23.554710211Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.004497185,\"lastEvaluation\":\"2024-03-27T08:45:23.550206212Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001992705,\"lastEvaluation\":\"2024-03-27T08:45:23.546625429Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.005092881,\"lastEvaluation\":\"2024-03-27T08:45:23.546619375Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-apps\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeContainerWaiting\",\"query\":\"sum by (namespace, pod, container, cluster) (kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"pod/{{ $labels.pod }} in namespace {{ $labels.namespace }} on container {{ $labels.container}} has been in waiting state for longer than 1 hour.\",\"summary\":\"Pod container waiting longer than 1 hour\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002153012,\"lastEvaluation\":\"2024-03-27T08:45:47.355995377Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetMisScheduled\",\"query\":\"kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.\",\"summary\":\"DaemonSet pods are misscheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001475897,\"lastEvaluation\":\"2024-03-27T08:45:47.360560273Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetNotScheduled\",\"query\":\"kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} - kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.\",\"summary\":\"DaemonSet pods are not scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002405728,\"lastEvaluation\":\"2024-03-27T08:45:47.358151446Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetRolloutStuck\",\"query\":\"((kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != 0) or (kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} has not finished or progressed for at least 30 minutes.\",\"summary\":\"DaemonSet rollout is stuck.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.009939706,\"lastEvaluation\":\"2024-03-27T08:45:47.346052138Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentGenerationMismatch\",\"query\":\"kube_deployment_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_deployment_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.\",\"summary\":\"Deployment generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006676917,\"lastEvaluation\":\"2024-03-27T08:45:47.318742591Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentRolloutStuck\",\"query\":\"kube_deployment_status_condition{condition=\\\"Progressing\\\",job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",status=\\\"false\\\"} != 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Rollout of deployment {{ $labels.namespace }}/{{ $labels.deployment }} is not progressing for longer than 15 minutes.\",\"summary\":\"Deployment rollout is not progressing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002685861,\"lastEvaluation\":\"2024-03-27T08:45:47.325423617Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaMaxedOut\",\"query\":\"kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has been running at max replicas for longer than 15 minutes.\",\"summary\":\"HPA is running at max replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001748196,\"lastEvaluation\":\"2024-03-27T08:45:47.372211016Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaReplicasMismatch\",\"query\":\"(kube_horizontalpodautoscaler_status_desired_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_horizontalpodautoscaler_spec_min_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003c kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and changes(kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[15m]) == 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has not matched the desired number of replicas for longer than 15 minutes.\",\"summary\":\"HPA has not matched desired number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.006123019,\"lastEvaluation\":\"2024-03-27T08:45:47.366084912Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobFailed\",\"query\":\"kube_job_failed{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeJobFailed.md\",\"summary\":\"Job failed to complete.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000962067,\"lastEvaluation\":\"2024-03-27T08:45:47.36511672Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobNotCompleted\",\"query\":\"time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} and kube_job_status_active{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0) \\u003e 43200\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than {{ \\\"43200\\\" | humanizeDuration }} to complete.\",\"summary\":\"Job did not complete in time\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003074286,\"lastEvaluation\":\"2024-03-27T08:45:47.362039128Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodCrashLooping\",\"query\":\"max_over_time(kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",reason=\\\"CrashLoopBackOff\\\"}[5m]) \\u003e= 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is in waiting state (reason: \\\"CrashLoopBackOff\\\").\",\"summary\":\"Pod is crash looping.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001448454,\"lastEvaluation\":\"2024-03-27T08:45:47.274928908Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodNotReady\",\"query\":\"sum by (namespace, pod, cluster) (max by (namespace, pod, cluster) (kube_pod_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Pending|Unknown\\\"} unless ignoring (phase) (kube_pod_status_unschedulable{job=\\\"kube-state-metrics\\\"} == 1)) * on (namespace, pod, cluster) group_left (owner_kind) topk by (namespace, pod, cluster) (1, max by (namespace, pod, owner_kind, cluster) (kube_pod_owner{owner_kind!=\\\"Job\\\"}))) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePodNotReady.md\",\"summary\":\"Pod has been in a non-ready state for more than 15 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.042353807,\"lastEvaluation\":\"2024-03-27T08:45:47.276380959Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetGenerationMismatch\",\"query\":\"kube_statefulset_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.\",\"summary\":\"StatefulSet generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003681367,\"lastEvaluation\":\"2024-03-27T08:45:47.333481829Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetReplicasMismatch\",\"query\":\"(kube_statefulset_status_replicas_ready{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[10m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.\",\"summary\":\"StatefulSet has not matched the expected number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.005360424,\"lastEvaluation\":\"2024-03-27T08:45:47.328115341Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetUpdateNotRolledOut\",\"query\":\"(max without (revision) (kube_statefulset_status_current_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} unless kube_statefulset_status_update_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) * (kube_statefulset_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.\",\"summary\":\"StatefulSet update has not been rolled out.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.008858762,\"lastEvaluation\":\"2024-03-27T08:45:47.337166375Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.099051569,\"lastEvaluation\":\"2024-03-27T08:45:47.27491246Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-resources\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-449b63f6-6347-432f-bc85-abb57ae71c00.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeCPUOvercommit\",\"query\":\"sum by (cluster) (namespace_cpu:kube_pod_container_resource_requests:sum{job=\\\"kube-state-metrics\\\"}) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted CPU resource requests for Pods by {{ $value }} CPU shares and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted CPU resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000825829,\"lastEvaluation\":\"2024-03-27T08:45:25.808313079Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeMemoryOvercommit\",\"query\":\"sum by (cluster) (namespace_memory:kube_pod_container_resource_requests:sum) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted memory resource requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted memory resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002568247,\"lastEvaluation\":\"2024-03-27T08:45:25.810491086Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaAlm
Ginkgo exit error 1: exit with code 1
periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-heavy-build-ovn-remote-libvirt-s390x (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1772881388521918464junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m26s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m26s, firing for 0s:
Mar 27 07:47:45.674 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 07:49:15.674 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-operator-framework-olm-master-e2e-upgrade (all) - 5 runs, 0% failed, 20% of runs match
#1772864236100259840junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h4m52s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h4m52s, firing for 0s:
Mar 27 06:45:54.398 - 716s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 06:45:54.398 - 3176s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-etcd-operator-release-4.14-e2e-aws-ovn-serial (all) - 1 runs, 0% failed, 100% of runs match
#1772853632237047808junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m28s, firing for 0s:
Mar 27 05:51:11.525 - 388s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-network-migration (all) - 1 runs, 0% failed, 100% of runs match
#1772856951936389120junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m4s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 1m4s, firing for 0s:
Mar 27 06:35:23.811 - 64s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-etcd-operator-release-4.14-e2e-azure-ovn-etcd-scaling (all) - 1 runs, 0% failed, 100% of runs match
#1772853632299962368junit31 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m56s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m56s, firing for 0s:
Mar 27 06:30:54.746 - 418s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:39:24.746 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-osde2e-main-nightly-4.16-conformance-osd-aws (all) - 2 runs, 100% failed, 50% of failures match = 50% impact
#1772851200006295552junit32 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772851200006295552junit32 hours ago
        <*errors.errorString | 0xc001984120>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"UpgradeConfigSyncFailureOver4HrSRE\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-monitoring\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"critical\"\n    },\n    \"value\": [\n      1711518791.533,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-heavy-build-ovn-remote-libvirt-s390x (all) - 2 runs, 0% failed, 50% of runs match
#1772866294853406720junit32 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"libvirt", Architecture:"s390x", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 27 06:37:07.201 - 58s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-etcd-operator-release-4.14-e2e-aws-etcd-recovery (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772853632174133248junit32 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 43m16s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 33m24s, firing for 9m52s:
Mar 27 06:14:01.375 - 1032s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 06:15:01.375 - 972s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 06:31:13.375 - 592s  E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above info
KubeAPIErrorBudgetBurn was at or above info for at least 9m52s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 33m24s, firing for 9m52s:
periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-cpu-partitioning (all) - 8 runs, 100% failed, 25% of failures match = 25% impact
#1772852422255841280junit32 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 22s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 22s, firing for 0s:
Mar 27 05:40:53.393 - 22s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772761824161173504junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 18m20s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 16m52s, firing for 1m28s:
Mar 26 23:45:15.494 - 4s    I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 23:45:15.494 - 4s    I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:46:47.494 - 232s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 23:46:47.494 - 772s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:45:19.494 - 88s   E alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-heavy-build-ovn-remote-libvirt-ppc64le (all) - 2 runs, 0% failed, 50% of runs match
#1772866294018740224junit32 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m2s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"libvirt", Architecture:"ppc64le", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m2s, firing for 0s:
Mar 27 06:31:04.595 - 302s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.16-e2e-gcp-sdn (all) - 7 runs, 0% failed, 14% of runs match
#1772842865584508928junit32 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m42s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 5m42s, firing for 0s:
Mar 27 05:07:18.765 - 284s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:08:34.765 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-ovn-upgrade-into-change (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
#1772845119389569024junit33 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m26s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m26s, firing for 0s:
Mar 27 05:32:18.140 - 206s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-cloud-credential-operator-release-4.16-periodics-e2e-azure-manual-oidc (all) - 2 runs, 0% failed, 50% of runs match
#1772820460904386560junit33 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 27 04:06:11.975 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-aws-ovn-upgrade (all) - 3 runs, 0% failed, 67% of runs match
#1772810519082700800junit33 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 50s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 50s, firing for 0s:
Mar 27 02:58:23.055 - 50s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772632233971027968junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h25m6s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h25m6s, firing for 0s:
Mar 26 15:09:15.240 - 828s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 15:09:15.240 - 4278s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-upgrade-from-stable-4.12-e2e-metal-ipi-upgrade-ovn-ipv6 (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
#1772798371799502848junit33 hours ago
Mar 27 05:03:45.277 - 486s  E clusteroperator/authentication condition/Degraded status/True reason/APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver ()\nOAuthServerDeploymentDegraded: 1 of 3 requested instances are unavailable for oauth-openshift.openshift-authentication ()
Mar 27 05:03:46.046 - 58s   E alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
Mar 27 05:04:16.775 E ns/openshift-ovn-kubernetes pod/ovnkube-node-s6lwj node/master-2.ostest.test.metalkube.org uid/a5f602f9-5dd3-4d03-93d5-a9105f870eec container/ovn-acl-logging reason/ContainerExit code/143 cause/Error
#1772798371799502848junit33 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 53m30s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 52m32s, firing for 58s:
Mar 27 04:47:42.046 - 964s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 04:58:42.046 - 304s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 05:00:16.046 - 210s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 27 05:04:44.046 - 558s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 05:04:44.046 - 558s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 27 05:04:44.046 - 558s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 27 05:03:46.046 - 58s   E alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-aws-sdn-upgrade (all) - 3 runs, 33% failed, 200% of failures match = 67% impact
#1772810518994620416junit34 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 42s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 42s, firing for 0s:
Mar 27 02:50:17.222 - 42s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772632233887141888junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 50s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 50s, firing for 0s:
Mar 26 15:03:45.198 - 50s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-aws-sdn-serial (all) - 4 runs, 25% failed, 200% of failures match = 50% impact
#1772810519174975488junit34 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m26s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 5m26s, firing for 0s:
Mar 27 02:52:53.467 - 326s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772632233824227328junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 44s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 44s, firing for 0s:
Mar 26 15:08:21.469 - 44s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-gcp-sdn (all) - 3 runs, 0% failed, 67% of runs match
#1772810519208529920junit34 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m0s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 12m0s, firing for 0s:
Mar 27 03:02:56.430 - 92s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 27 03:02:56.430 - 164s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 03:02:56.430 - 464s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772632233866170368junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m46s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m46s, firing for 0s:
Mar 26 15:17:08.316 - 258s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 15:17:58.316 - 28s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
pull-ci-openshift-api-master-e2e-aws-serial (all) - 29 runs, 28% failed, 25% of failures match = 7% impact
#1772796590373736448junit35 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h0m54s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h0m54s, firing for 0s:
Mar 27 02:09:41.246 - 432s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 02:09:41.246 - 3222s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772703663073005568junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 25m42s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 25m42s, firing for 0s:
Mar 26 19:56:58.973 - 186s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 19:56:58.973 - 1356s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-sdn-serial (all) - 4 runs, 0% failed, 25% of runs match
#1772798366762143744junit35 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m22s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m22s, firing for 0s:
Mar 27 02:02:12.545 - 202s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-vsphere-ovn-serial (all) - 4 runs, 0% failed, 25% of runs match
#1772798388580913152junit35 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m12s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m12s, firing for 0s:
Mar 27 02:25:12.391 - 72s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-aws-sdn-techpreview-serial (all) - 4 runs, 0% failed, 25% of runs match
#1772798374324473856junit35 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m38s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m38s, firing for 0s:
Mar 27 02:03:02.836 - 98s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-azure-ovn (all) - 4 runs, 0% failed, 25% of runs match
#1772798390254440448junit35 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m44s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m44s, firing for 0s:
Mar 27 02:24:50.463 - 164s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-vsphere-ovn-upi-serial (all) - 4 runs, 50% failed, 100% of failures match = 50% impact
#1772798368037212160junit35 hours ago
        <*errors.errorString | 0xc00219a740>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711505203.464,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711505203.464,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711505203.464,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711505203.464,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711505203.464,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711505203.464,\n      \"1\"\n    ]\n  }\n]",
        },
#1772798368037212160junit35 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
#1772617076507152384junit47 hours ago
        <*errors.errorString | 0xc00271b0e0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711461863.485,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711461863.485,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesNorthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711461863.485,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711461863.485,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711461863.485,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711461863.485,\n      \"1\"\n    ]\n  }\n]",
        },
#1772617076507152384junit47 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
periodic-ci-openshift-release-master-nightly-4.13-e2e-gcp-sdn (all) - 4 runs, 0% failed, 50% of runs match
#1772798380188110848junit35 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m12s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m12s, firing for 0s:
Mar 27 02:10:42.218 - 72s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772617121738526720junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 30s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 30s, firing for 0s:
Mar 26 14:05:21.150 - 30s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-csi (all) - 3 runs, 0% failed, 33% of runs match
#1772798335288086528junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m48s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m48s, firing for 0s:
Mar 27 02:09:23.088 - 228s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-ingress-operator-master-e2e-azure-ovn (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
#1772777122989871104junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 01:17:31.209 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-operator-framework-olm-master-periodics-e2e-gcp-ovn (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1772775778337951744junit36 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m16s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 12m16s, firing for 0s:
Mar 27 01:00:56.885 - 188s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 27 01:00:56.885 - 548s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-azure-ovn-shared-vpc (all) - 11 runs, 9% failed, 100% of failures match = 9% impact
#1772751629062246400junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 27 00:31:22.749 - 88s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64-techpreview-serial (all) - 4 runs, 0% failed, 25% of runs match
#1772763352901095424junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m10s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m10s, firing for 0s:
Mar 26 23:48:09.346 - 250s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.12-ocp-e2e-serial-aws-ovn-heterogeneous (all) - 2 runs, 0% failed, 50% of runs match
#1772765347229732864junit37 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m40s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 6m40s, firing for 0s:
Mar 27 00:02:01.737 - 400s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
aggregator-periodic-ci-openshift-release-master-nightly-4.16-e2e-metal-ipi-serial-ovn-ipv6 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772726008210788352junit38 hours ago
# openshift-tests.[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
name: '[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at
  or above pending'
#1772726008210788352junit38 hours ago
# openshift-tests.[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above info
name: '[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at
  or above info'
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-upgrade-aws-ovn-arm64 (all) - 4 runs, 0% failed, 25% of runs match
#1772763353001758720junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m12s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m12s, firing for 0s:
Mar 26 23:47:22.096 - 192s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64 (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1772763352796237824junit38 hours ago
2024-03-27T00:35:04Z: Call to sippy finished after: 3.611086555s
response Body: {"ProwJobName":"periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64","ProwJobRunID":1772763352796237824,"Release":"4.13","CompareRelease":"4.13","Tests":[{"Name":"[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above info","Risk":{"Level":{"Name":"High","Level":100},"Reasons":["This test has passed 100.00% of 13 runs on jobs ['periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64'] in the last 14 days."]},"OpenBugs":[]},{"Name":"[sig-node] static pods should start after being created","Risk":{"Level":{"Name":"High","Level":100},"Reasons":["This test has passed 100.00% of 13 runs on jobs ['periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64'] in the last 14 days."]},"OpenBugs":[]}],"OverallRisk":{"Level":{"Name":"High","Level":100},"Reasons":["Maximum failed test risk: High"]},"OpenBugs":[]}
#1772763352796237824junit38 hours ago
2024-03-27T00:35:04Z: Call to sippy finished after: 3.611086555s
response Body: {"ProwJobName":"periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64","ProwJobRunID":1772763352796237824,"Release":"4.13","CompareRelease":"4.13","Tests":[{"Name":"[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above info","Risk":{"Level":{"Name":"High","Level":100},"Reasons":["This test has passed 100.00% of 13 runs on jobs ['periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64'] in the last 14 days."]},"OpenBugs":[]},{"Name":"[sig-node] static pods should start after being created","Risk":{"Level":{"Name":"High","Level":100},"Reasons":["This test has passed 100.00% of 13 runs on jobs ['periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64'] in the last 14 days."]},"OpenBugs":[]}],"OverallRisk":{"Level":{"Name":"High","Level":100},"Reasons":["Maximum failed test risk: High"]},"OpenBugs":[]}
#1772763352796237824junit38 hours ago
Mar 26 23:53:04.423 - 94s   E alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 26 23:53:04.423 - 2474s E alert/Watchdog ns/openshift-monitoring ALERTS{alertname="Watchdog", alertstate="firing", namespace="openshift-monitoring", prometheus="openshift-monitoring/k8s", severity="none"}
#1772763352796237824junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 26m40s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 25m6s, firing for 1m34s:
Mar 26 23:54:38.423 - 168s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 23:54:38.423 - 1338s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 23:53:04.423 - 94s   E alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-metal-ipi-upgrade-ovn-ipv6 (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
#1772707746769014784junit38 hours ago
        <*errors.errorString | 0xc007eb4980>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"CsvAbnormalOver30Min\",\n      \"alertstate\": \"firing\",\n      \"container\": \"olm-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-operator-lifecycle-manager\",\n      \"instance\": \"[fd01:0:0:2::f]:8443\",\n      \"job\": \"olm-operator-metrics\",\n      \"name\": \"packageserver\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"phase\": \"Pending\",\n      \"pod\": \"olm-operator-5f96d75978-9trgn\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"RequirementsUnknown\",\n      \"service\": \"olm-operator-metrics\",\n      \"severity\": \"warning\",\n      \"version\": \"0.19.0\"\n    },\n    \"value\": [\n      1711489457.098,\n      \"1\"\n    ]\n  }\n]",
        },
#1772707746769014784junit38 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
periodic-ci-openshift-release-master-ci-4.14-e2e-aws-sdn-techpreview-serial (all) - 5 runs, 40% failed, 50% of failures match = 20% impact
#1772730499362459648junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m36s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m36s, firing for 0s:
Mar 26 22:12:59.583 - 276s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-aws-efs-csi-driver-operator-master-nightly-operator-e2e (all) - 2 runs, 0% failed, 50% of runs match
#1772759055522074624junit38 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m58s, firing for 0s:
Mar 26 23:43:52.696 - 358s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
rehearse-46560-periodic-ci-openshift-multiarch-master-nightly-4.15-ocp-assisted-sno-power (all) - 3 runs, 100% failed, 33% of failures match = 33% impact
#1772721477901094912junit39 hours ago
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:108]: Failed to fetch alerting rules: unable to query https://thanos-querier-openshift-monitoring.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/api/v1/rules: host command failed: exit status 28
{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000237293,"lastEvaluation":"2024-03-26T23:19:31.249487072Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.00025617,"lastEvaluation":"2024-03-26T23:19:31.249229299Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000282678,"lastEvaluation":"2024-03-26T23:19:31.248944877Z","type":"alerting"}],"interval":30,"evaluationTime":0.000797209,"lastEvaluation":"2024-03-26T23:19:31.248929684Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.005031864,"lastEvaluation":"2024-03-26T23:19:31.395949072Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.007494008,"lastEvaluation":"2024-03-26T23:19:31.409054741Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.008061362,"lastEvaluation":"2024-03-26T23:19:31.400983496Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000133911,"lastEvaluation":"2024-03-26T23:19:31.416550473Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000292721,"lastEvaluation":"2024-03-26T23:19:31.416685998Z","type":"alerting"}],"interval":30,"evaluationTime":0.021042873,"lastEvaluation":"2024-03-26T23:19:31.395938586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000256886,"lastEvaluation":"2024-03-26T23:19:33.670339275Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.000935746,"lastEvaluation":"2024-03-26T23:19:33.670598709Z","type":"alerting"}],"interval":30,"evaluationTime":0.001212136,"lastEvaluation":"2024-03-26T23:19:33.670325705Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000286588,"lastEvaluation":"2024-03-26T23:19:33.526557915Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000473896,"lastEvaluation":"2024-03-26T23:19:33.526081372Z","type":"alerting"}],"interval":30,"evaluationTime":0.000779206,"lastEvaluation":"2024-03-26T23:19:33.526068667Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000349221,"lastEvaluation":"2024-03-26T23:19:30.563536654Z","type":"alerting"}],"interval":30,"e{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000256886,"lastEvaluation":"2024-03-26T23:19:33.670339275Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.000935746,"lastEvaluation":"2024-03-26T23:19:33.670598709Z","type":"alerting"}],"interval":30,"evaluationTime":0.001212136,"lastEvaluation":"2024-03-26T23:19:33.670325705Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000286588,"lastEvaluation":"2024-03-26T23:19:33.526557915Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000473896,"lastEvaluation":"2024-03-26T23:19:33.526081372Z","type":"alerting"}],"interval":30,"evaluationTime":0.000779206,"lastEvaluation":"2024-03-26T23:19:33.526068667Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluati{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluatio{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluatio{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000395125,"lastEvaluation":"2024-03-26T23:20:10.034645897Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000572266,"lastEvaluation":"2024-03-26T23:20:10.030396967Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.00162457,"lastEvaluation":"2024-03-26T23:20:10.031935043Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000956651,"lastEvaluation":"2024-03-26T23:20:10.030974917Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.001077533,"lastEvaluation":"2024-03-26T23:20:10.033563948Z","type":"alerting"}],"interval":30,"evaluationTime":0.004665296,"lastEvaluation":"2024-03-26T23:20:10.030380586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"2024-03-26T23:20:08.899763907Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.001156355,"lastEvaluation":"2024-03-26T23:20:08.898603972Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000748404,"lastEvaluation":"2024-03-26T23:20:08.897850749Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000380934,"lastEvaluation":"2024-03-26T23:20:08.897455708Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000347778,"lastEvaluation":"2024-03-26T23:20:08.897103847Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.001301448,"lastEvaluation":"2024-03-26T23:20:08.895511424Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000275968,"lastEvaluation":"2024-03-26T23:20:08.896819204Z","type":"alerting"}],"interval":30,"evaluationTime":0.006396776,"lastEvaluation":"2024-03-26T23:20:08.895492559Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000573366,"lastEvaluation":"2024-03-26T23:20:11.819467845Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000760536,"lastEvaluation":"2024-03-26T23:20:11.818702105Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000514415,"lastEvaluation":"2024-03-26T23:20:11.820045564Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000457601,"lastEvaluation":"2024-03-26T23:20:11.820565464Z","type":"recording"}],"interval":30,"evaluationTime":0.002342689,"lastEvaluation":"2024-03-26T23:20:11.818685752Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is
Ginkgo exit error 1: exit with code 1
#1772721477901094912junit39 hours ago
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:108]: Failed to fetch alerting rules: unable to query https://thanos-querier-openshift-monitoring.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/api/v1/rules: host command failed: exit status 28
{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000237293,"lastEvaluation":"2024-03-26T23:19:31.249487072Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.00025617,"lastEvaluation":"2024-03-26T23:19:31.249229299Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000282678,"lastEvaluation":"2024-03-26T23:19:31.248944877Z","type":"alerting"}],"interval":30,"evaluationTime":0.000797209,"lastEvaluation":"2024-03-26T23:19:31.248929684Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.005031864,"lastEvaluation":"2024-03-26T23:19:31.395949072Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.007494008,"lastEvaluation":"2024-03-26T23:19:31.409054741Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.008061362,"lastEvaluation":"2024-03-26T23:19:31.400983496Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000133911,"lastEvaluation":"2024-03-26T23:19:31.416550473Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000292721,"lastEvaluation":"2024-03-26T23:19:31.416685998Z","type":"alerting"}],"interval":30,"evaluationTime":0.021042873,"lastEvaluation":"2024-03-26T23:19:31.395938586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000256886,"lastEvaluation":"2024-03-26T23:19:33.670339275Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.000935746,"lastEvaluation":"2024-03-26T23:19:33.670598709Z","type":"alerting"}],"interval":30,"evaluationTime":0.001212136,"lastEvaluation":"2024-03-26T23:19:33.670325705Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000286588,"lastEvaluation":"2024-03-26T23:19:33.526557915Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000473896,"lastEvaluation":"2024-03-26T23:19:33.526081372Z","type":"alerting"}],"interval":30,"evaluationTime":0.000779206,"lastEvaluation":"2024-03-26T23:19:33.526068667Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluat{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000256886,"lastEvaluation":"2024-03-26T23:19:33.670339275Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.000935746,"lastEvaluation":"2024-03-26T23:19:33.670598709Z","type":"alerting"}],"interval":30,"evaluationTime":0.001212136,"lastEvaluation":"2024-03-26T23:19:33.670325705Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluati{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluation{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"2024-03-26T23:20:08.899763907Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.001156355,"lastEvaluation":"2024-03-26T23:20:08.898603972Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000748404,"lastEvaluation":"2024-03-26T23:20:08.897850749Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000380934,"lastEvaluation":"2024-03-26T23:20:08.897455708Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000347778,"lastEvaluation":"2024-03-26T23:20:08.897103847Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.001301448,"lastEvaluation":"2024-03-26T23:20:08.895511424Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000275968,"lastEvaluation":"2024-03-26T23:20:08.896819204Z","type":"alerting"}],"interval":30,"evaluationTime":0.006396776,"lastEvaluation":"2024-03-26T23:20:08.895492559Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"20{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000395125,"lastEvaluation":"2024-03-26T23:20:10.034645897Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000572266,"lastEvaluation":"2024-03-26T23:20:10.030396967Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.00162457,"lastEvaluation":"2024-03-26T23:20:10.031935043Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000956651,"lastEvaluation":"2024-03-26T23:20:10.030974917Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.001077533,"lastEvaluation":"2024-03-26T23:20:10.033563948Z","type":"alerting"}],"interval":30,"evaluationTime":0.004665296,"lastEvaluation":"2024-03-26T23:20:10.030380586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"2024-03-26T23:20:08.899763907Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.001156355,"lastEvaluation":"2024-03-26T23:20:08.898603972Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000748404,"lastEvaluation":"2024-03-26T23:20:08.897850749Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000380934,"lastEvaluation":"2024-03-26T23:20:08.897455708Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000347778,"lastEvaluation":"2024-03-26T23:20:08.897103847Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.001301448,"lastEvaluation":"2024-03-26T23:20:08.895511424Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000275968,"lastEvaluation":"2024-03-26T23:20:08.896819204Z","type":"alerting"}],"interval":30,"evaluationTime":0.006396776,"lastEvaluation":"2024-03-26T23:20:08.895492559Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000573366,"lastEvaluation":"2024-03-26T23:20:11.819467845Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000760536,"lastEvaluation":"2024-03-26T23:20:11.818702105Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000514415,"lastEvaluation":"2024-03-26T23:20:11.820045564Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000457601,"lastEvaluation":"2024-03-26T23:20:11.820565464Z","type":"recording"}],"interval":30,"evaluationTime":0.002342689,"lastEvaluation":"2024-03-26T23:20:11.818685752Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:
Ginkgo exit error 1: exit with code 1
#1772721477901094912junit39 hours ago
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:108]: Failed to fetch alerting rules: unable to query https://thanos-querier-openshift-monitoring.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/api/v1/rules: host command failed: exit status 28
{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000237293,"lastEvaluation":"2024-03-26T23:19:31.249487072Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.00025617,"lastEvaluation":"2024-03-26T23:19:31.249229299Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000282678,"lastEvaluation":"2024-03-26T23:19:31.248944877Z","type":"alerting"}],"interval":30,"evaluationTime":0.000797209,"lastEvaluation":"2024-03-26T23:19:31.248929684Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.005031864,"lastEvaluation":"2024-03-26T23:19:31.395949072Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.007494008,"lastEvaluation":"2024-03-26T23:19:31.409054741Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.008061362,"lastEvaluation":"2024-03-26T23:19:31.400983496Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000133911,"lastEvaluation":"2024-03-26T23:19:31.416550473Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000292721,"lastEvaluation":"2024-03-26T23:19:31.416685998Z","type":"alerting"}],"interval":30,"evaluationTime":0.021042873,"lastEvaluation":"2024-03-26T23:19:31.395938586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000256886,"lastEvaluation":"2024-03-26T23:19:33.670339275Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.000935746,"lastEvaluation":"2024-03-26T23:19:33.670598709Z","type":"alerting"}],"interval":30,"evaluationTime":0.001212136,"lastEvaluation":"2024-03-26T23:19:33.670325705Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000286588,"lastEvaluation":"2024-03-26T23:19:33.526557915Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000473896,"lastEvaluation":"2024-03-26T23:19:33.526081372Z","type":"alerting"}],"interval":30,"evaluationTime":0.000779206,"lastEvaluation":"2024-03-26T23:19:33.526068667Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"20{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000256886,"lastEvaluation":"2024-03-26T23:19:33.670339275Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.000935746,"lastEvaluation":"2024-03-26T23:19:33.670598709Z","type":"alerting"}],"interval":30,"evaluationTime":0.001212136,"lastEvaluation":"2024-03-26T23:19:33.670325705Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000286588,"lastEvaluation":"2024-03-26T23:19:33.526557915Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000473896,"lastEvaluation":"2024-03-26T23:19:33.526081372Z","type":"alerting"}],"interval":30,"evaluationTime":0.000779206,"lastEvaluation":"2024-03-26T23:19:33.526068667Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"ev{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluationTime":0.000812595,"lastEvaluation":"2024-03-26T23:20:00.563494815Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-without-valid-node-ref","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithoutValidNode","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"If the machine never became a node, you should diagnose the machine related failures.\nIf the node was deleted from the API, you may delete the machine if appropriate.\n","summary":"machine {{ $labels.name }} does not have valid node reference"},"alerts":[],"health":"ok","evaluationTime":0.000754463,"lastEvaluation":"2024-03-26T23:19:56.39608856Z","type":"alerting"}],"interval":30,"evaluationTime":0.000780961,"lastEvaluation":"2024-03-26T23:19:56.396069777Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcc-drain-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"state":"inactive","name":"MCCDrainError","query":"mcc_drain_err \u003e 0","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller","summary":"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times."},"alerts":[],"health":"ok","evaluationTime":0.000527222,"lastEvaluation":"2024-03-26T23:19:50.406498096Z","type":"alerting"}],"interval":30,"evaluationTime":0.000552065,"lastEvaluation":"2024-03-26T23:19:50.406479966Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcc-pool-alert","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"state":"inactive","name":"MCCPoolAlert","query":"mcc_pool_alert \u003e 0","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller","summary":"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored."},"alerts":[],"health":"ok","evaluationTime":0.000187879,"lastEvaluation":"2024-03-26T23:19:46.287841599Z","type":"alerting"}],"interval":30,"evaluationTime":0.00019628,"lastEvaluation":"2024-03-26T23:19:46.287835828Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"os-image-override.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"name":"os_image_url_override:sum","query":"sum(os_image_url_override)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000383984,"lastEvaluation":"2024-03-26T23:19:44.314511353Z","type":"recording"}],"interval":30,"evaluationTime":0.000395439,"lastEvaluation":"2024-03-26T23:19:44.31450672Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"extremely-high-individual-control-plane-memory","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneMemory","query":"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\"))) * 100 \u003e 90","duration":2700,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.","summary":"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability."},"alerts":[],"health":"ok","evaluationTime":0.000903038,"lastEvaluation":"2024-03-26T23:19:42.933251215Z","type":"alerting"}],"interval":30,"evaluationTime":0.000919296,"lastEvaluation":"2024-03-26T23:19:42.933238883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"high-overall-control-plane-memory","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"HighOverallControlPlaneMemory","query":"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\"))) * 100 \u003e 60","duration":3600,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.","summary":"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability."},"alerts":[],"health":"ok","evaluationTime":0.00067969,"lastEvaluation":"2024-03-26T23:19:40.598102663Z","type":"alerting"}],"interval":30,"evaluationTime":0.000694461,"lastEvaluation":"2024-03-26T23:19:40.598090936Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-kubelet-health-state-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"KubeletHealthState","query":"mcd_kubelet_state \u003e 2","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Kubelet health failure threshold reached","summary":"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur."},"alerts":[],"health":"ok","evaluationTime":0.000357465,"lastEvaluation":"2024-03-26T23:19:37.417665533Z","type":"alerting"}],"interval":30,"evaluationTime":0.000377872,"lastEvaluation":"2024-03-26T23:19:37.417649377Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-pivot-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"MCDPivotError","query":"mcd_pivot_errors_total \u003e 0","duration":120,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon ","summary":"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000288399,"lastEvaluation":"2024-03-26T23:19:40.50170944Z","type":"alerting"}],"interval":30,"evaluationTime":0.000305165,"lastEvaluation":"2024-03-26T23:19:40.501698198Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-reboot-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"MCDRebootError","query":"mcd_reboots_failed_total \u003e 0","duration":300,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon ","summary":"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000299106,"lastEvaluation":"2024-03-26T23:19:47.605849339Z","type":"alerting"}],"interval":30,"evaluationTime":0.000319047,"lastEvaluation":"2024-03-26T23:19:47.605834017Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"system-memory-exceeds-reservation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"firing","name":"SystemMemoryExceedsReservation","query":"sum by (node) (container_memory_rss{id=\"/system.slice\"}) \u003e ((sum by (node) (kube_node_status_capacity{resource=\"memory\"} - kube_node_status_allocatable{resource=\"memory\"})) * 0.95)","duration":900,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).","summary":"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved"},"alerts":[{"labels":{"alertname":"SystemMemoryExceedsReservation","namespace":"openshift-machine-config-operator","node":"fa-c2-39-02-35-20","severity":"warning"},"annotations":{"description":"System memory usage of 1.18G on fa-c2-39-02-35-20 exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).","summary":"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved"},"state":"firing","activeAt":"2024-03-26T21:32:38.8916566Z","value":"1.179844608e+09","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00051687,"lastEvaluation":"20{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"2024-03-26T23:20:08.899763907Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.001156355,"lastEvaluation":"2024-03-26T23:20:08.898603972Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000748404,"lastEvaluation":"2024-03-26T23:20:08.897850749Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000380934,"lastEvaluation":"2024-03-26T23:20:08.897455708Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000347778,"lastEvaluation":"2024-03-26T23:20:08.897103847Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.001301448,"lastEvaluation":"2024-03-26T23:20:08.895511424Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000275968,"lastEvaluation":"2024-03-26T23:20:08.896819204Z","type":"alerting"}],"interval":30,"evaluationTime":0.006396776,"lastEvaluation":"2024-03-26T23:20:08.895492559Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluat{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000395125,"lastEvaluation":"2024-03-26T23:20:10.034645897Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000572266,"lastEvaluation":"2024-03-26T23:20:10.030396967Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.00162457,"lastEvaluation":"2024-03-26T23:20:10.031935043Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000956651,"lastEvaluation":"2024-03-26T23:20:10.030974917Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.001077533,"lastEvaluation":"2024-03-26T23:20:10.033563948Z","type":"alerting"}],"interval":30,"evaluationTime":0.004665296,"lastEvaluation":"2024-03-26T23:20:10.030380586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"2024-03-26T23:20:08.899763907Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.001156355,"lastEvaluation":"2024-03-26T23:20:08.898603972Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000748404,"lastEvaluation":"2024-03-26T23:20:08.897850749Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000380934,"lastEvaluation":"2024-03-26T23:20:08.897455708Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000347778,"lastEvaluation":"2024-03-26T23:20:08.897103847Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.001301448,"lastEvaluation":"2024-03-26T23:20:08.895511424Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000275968,"lastEvaluation":"2024-03-26T23:20:08.896819204Z","type":"alerting"}],"interval":30,"evaluationTime":0.006396776,"lastEvaluation":"2024-03-26T23:20:08.895492559Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd
Ginkgo exit error 1: exit with code 1
#1772721477901094912junit39 hours ago
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:108]: Failed to fetch alerting rules: unable to query https://thanos-querier-openshift-monitoring.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/api/v1/rules: host command failed: exit status 28
{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.005031864,"lastEvaluation":"2024-03-26T23:19:31.395949072Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.007494008,"lastEvaluation":"2024-03-26T23:19:31.409054741Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.008061362,"lastEvaluation":"2024-03-26T23:19:31.400983496Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000133911,"lastEvaluation":"2024-03-26T23:19:31.416550473Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000292721,"lastEvaluation":"2024-03-26T23:19:31.416685998Z","type":"alerting"}],"interval":30,"evaluationTime":0.021042873,"lastEvaluation":"2024-03-26T23:19:31.395938586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000256886,"lastEvaluation":"2024-03-26T23:19:33.670339275Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.000935746,"lastEvaluation":"2024-03-26T23:19:33.670598709Z","type":"alerting"}],"interval":30,"evaluationTime":0.001212136,"lastEvaluation":"2024-03-26T23:19:33.670325705Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000286588,"lastEvaluation":"2024-03-26T23:19:33.526557915Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000473896,"lastEvaluation":"2024-03-26T23:19:33.526081372Z","type":"alerting"}],"interval":30,"evaluationTime":0.000779206,"lastEvaluation":"2024-03-26T23:19:33.526068667Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluat{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluationTime":0.000812595,"lastEvaluation":"2024-03-26T23:20:00.563494815Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-without-valid-node-ref","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithoutValidNode","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"If the machine never became a node, you should diagnose the machine related failures.\nIf the node was deleted from the API, you may delete the machine if appropriate.\n","summary":"machine {{ $labels.name }} does not have valid node reference"},"alerts":[],"health":"ok","evaluationTime":0.000754463,"lastEvaluation":"2024-03-26T23:19:56.39608856Z","type":"alerting"}],"interval":30,"evaluationTime":0.000780961,"lastEvaluation":"2024-03-26T23:19:56.396069777Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcc-drain-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"state":"inactive","name":"MCCDrainError","query":"mcc_drain_err \u003e 0","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller","summary":"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times."},"alerts":[],"health":"ok","evaluationTime":0.000527222,"lastEvaluation":"2024-03-26T23:19:50.406498096Z","type":"alerting"}],"interval":30,"evaluationTime":0.000552065,"lastEvaluation":"2024-03-26T23:19:50.406479966Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcc-pool-alert","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"state":"inactive","name":"MCCPoolAlert","query":"mcc_pool_alert \u003e 0","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller","summary":"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored."},"alerts":[],"health":"ok","evaluationTime":0.000187879,"lastEvaluation":"2024-03-26T23:19:46.287841599Z","type":"alerting"}],"interval":30,"evaluationTime":0.00019628,"lastEvaluation":"2024-03-26T23:19:46.287835828Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"os-image-override.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"name":"os_image_url_override:sum","query":"sum(os_image_url_override)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000383984,"lastEvaluation":"2024-03-26T23:19:44.314511353Z","type":"recording"}],"interval":30,"evaluationTime":0.000395439,"lastEvaluation":"2024-03-26T23:19:44.31450672Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"extremely-high-individual-control-plane-memory","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneMemory","query":"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\"))) * 100 \u003e 90","duration":2700,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.","summary":"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability."},"alerts":[],"health":"ok","evaluationTime":0.000903038,"lastEvaluation":"2024-03-26T23:19:42.933251215Z","type":"alerting"}],"interval":30,"evaluationTime":0.000919296,"lastEvaluation":"2024-03-26T23:19:42.933238883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"high-overall-control-plane-memory","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"HighOverallControlPlaneMemory","query":"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\"))) * 100 \u003e 60","duration":3600,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.","summary":"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability."},"alerts":[],"health":"ok","evaluationTime":0.00067969,"lastEvaluation":"2024-03-26T23:19:40.598102663Z","type":"alerting"}],"interval":30,"evaluationTime":0.000694461,"lastEvaluation":"2024-03-26T23:19:40.598090936Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-kubelet-health-state-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"KubeletHealthState","query":"mcd_kubelet_state \u003e 2","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Kubelet health failure threshold reached","summary":"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur."},"alerts":[],"health":"ok","evaluationTime":0.000357465,"lastEvaluation":"2024-03-26T23:19:37.417665533Z","type":"alerting"}],"interval":30,"evaluationTime":0.000377872,"lastEvaluation":"2024-03-26T23:19:37.417649377Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-pivot-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"MCDPivotError","query":"mcd_pivot_errors_total \u003e 0","duration":120,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon ","summary":"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000288399,"lastEvaluation":"2024-03-26T23:19:40.50170944Z","type":"alerting"}],"interval":30,"evaluationTime":0.000305165,"lastEvaluation":"2024-03-26T23:19:40.501698198Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-reboot-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"MCDRebootError","query":"mcd_reboots_failed_total \u003e 0","duration":300,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon ","summary":"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000299106,"lastEvaluation":"2024-03-26T23:19:47.605849339Z","type":"alerting"}],"interval":30,"evaluationTime":0.000319047,"lastEvaluation":"2024-03-26T23:19:47.605834017Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"system-memory-exceeds-reservation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"firing","name":"SystemMemoryExceedsReservation","query":"sum by (node) (container_memory_rss{id=\"/system.slice\"}) \u003e ((sum by (node) (kube_node_status_capacity{resource=\"memory\"} - kube_node_status_allocatable{resource=\"memory\"})) * 0.95)","duration":900,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).","summary":"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved"},"alerts":[{"labels":{"alertname":"SystemMemoryExceedsReservation","namespace":"openshift-machine-config-operator","node":"fa-c2-39-02-35-20","severity":"warning"},"annotations":{"description":"System memory usage of 1.18G on fa-c2-39-02-35-20 exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).","summary":"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved"},"state":"firing","activeAt":"2024-03-26T21:32:38.8916566Z","value":"1.179844608e+09","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00051687,"lastEvaluation":"2024-03-26T23:19:38.900979259Z","type":"alerting"}],"interval":30,"evaluationTime":0.000521519,"lastEvaluation":"2024-03-26T23:19:38.900977333Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"operator.marketplace.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-marketplace-marketplace-alert-rules-bc1b3701-c646-4cca-b840-4027dd66696f.yaml","rules":[{"state":"inactive","name":"OperatorHubSourceError","query":"catalogsource_ready{exported_namespace=\"openshift-marketplace\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Operators shipped via the {{ $labels.name }} source are not available for installation until the issue is fixed. Operators already installed from this source will not receive updates until issue is fixed. Inspect the status of the pod owned by {{ $labels.name }} source in the openshift-marketplace namespace (oc -n openshift-marketplace get pods -l olm.catalogSource={{ $labels.name }}) to diagnose and repair.","summary":"The {{ $labels.name }} source is in non-ready state for more than 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000232192,"lastEvaluation":"2024-03-26T23:19:48.937113746Z","type":"alerting"}],"interval":30,"evaluationTime":0.000240996,"lastEvaluation":"2024-03-26T23:19:48.93710773Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"alertmanager.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-alertmanager-main-rules-83f7de78-ab8e-4679-9f89-1f5befba7380.yaml","rules":[{"state":"inactive","name":"AlertmanagerClusterDown","query":"(count by (namespace, service) (avg_over_time(up{job=~\"alertmanager-main|alertmanager-user-workload\"}[5m]) \u003c 0.5) / count by (namespace, service) (up{job=~\"alertmanager-main|alertmanager-user-workload\"})) \u003e= 0.5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value | humanizePercentage }} of Alertmanager instances within the {{$labels.job}} cluster have been up for less than half of the last 5m.","summary":"Half or more of the Alertmanager instances within the same cluster are down."},"alerts":[],"health":"ok","evaluationTime":0.000474207,"lastEvaluation":"2024-03-26T23:19:37.697791484Z","type":"alerting"},{"state":"inactive","name":"AlertmanagerClusterFailedToSendAlerts","query":"min by (namespace, service, integration) (rate(alertmanager_notifications_failed_total{integration=~\".*\",job=~\"alertmanager-main|alertmanager-user-workload\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{integration=~\".*\",job=~\"alertmanager-main|alertmanager-user-workload\"}[5m])) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The minimum notification failure rate to {{ $labels.integration }} sent from any instance in the {{$labels.job}} cluster is {{ $value | humanizePercentage }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerClusterFailedToSendAlerts.md","summary":"All Alertmanager instances in a cluster failed to send notifications to a critical integration."},"alerts":[],"health":"ok","evaluationTime":0.000959907,"lastEvaluation":"2024-03-26T23:19:37.696632356Z","type":"alerting"},{"state":"inactive","name":"AlertmanagerConfigInconsistent","query":"count by (namespace, service) (count_values by (namespace, service) (\"config_hash\", alertmanager_config_hash{job=~\"alertmanager-main|alertmanager-user-workload\"})) != 1","duration":1200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Alertmanager instances within the {{$labels.job}} cluster have different configurations.","summary":"Alertmanager instances within the same cluster have different configurations."},"alerts":[],"health":"ok","evaluationTime":0.00018893,"lastEvaluation":"2024-03-26T23:19:37.697593943Z","type":"alerting"},{"state":"inactive",{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000395125,"lastEvaluation":"2024-03-26T23:20:10.034645897Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000572266,"lastEvaluation":"2024-03-26T23:20:10.030396967Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.00162457,"lastEvaluation":"2024-03-26T23:20:10.031935043Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000956651,"lastEvaluation":"2024-03-26T23:20:10.030974917Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.001077533,"lastEvaluation":"2024-03-26T23:20:10.033563948Z","type":"alerting"}],"interval":30,"evaluationTime":0.004665296,"lastEvaluation":"2024-03-26T23:20:10.030380586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"2024-03-26T23:20:08.899763907Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.001156355,"lastEvaluation":"2024-03-26T23:20:08.898603972Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000748404,"lastEvaluation":"2024-03-26T23:20:08.897850749Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000380934,"lastEvaluation":"2024-03-26T23:20:08.897455708Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000347778,"lastEvaluation":"2024-03-26T23:20:08.897103847Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.001301448,"lastEvaluation":"2024-03-26T23:20:08.895511424Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000275968,"lastEvaluation":"2024-03-26T23:20:08.896819204Z","type":"alerting"}],"interval":30,"evaluationTime":0.006396776,"lastEvaluation":"2024-03-26T23:20:08.895492559Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evalua{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000139903,"lastEvaluation":"2024-03-26T23:20:12.635267096Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000164241,"lastEvaluation":"2024-03-26T23:20:12.63540953Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.00019769,"lastEvaluation":"2024-03-26T23:20:12.635061792Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000158249,"lastEvaluation":"2024-03-26T23:20:12.635576586Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000496007,"lastEvaluation":"2024-03-26T23:20:12.634561549Z","type":"alerting"}],"interval":30,"evaluationTime":0.001189226,"lastEvaluation":"2024-03-26T23:20:12.63455098Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000395125,"lastEvaluation":"2024-03-26T23:20:10.034645897Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000572266,"lastEvaluation":"2024-03-26T23:20:10.030396967Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.00162457,"lastEvaluation":"2024-03-26T23:20:10.031935043Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000956651,"lastEvaluation":"2024-03-26T23:20:10.030974917Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.001077533,"lastEvaluation":"2024-03-26T23:20:10.033563948Z","type":"alerting"}],"interval":30,"evaluationTime":0.004665296,"lastEvaluation":"2024-03-26T23:20:10.030380586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluati
Ginkgo exit error 1: exit with code 1
#1772721477901094912junit39 hours ago
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:108]: Failed to fetch alerting rules: unable to query https://thanos-querier-openshift-monitoring.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/api/v1/rules: host command failed: exit status 28
{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000408387,"lastEvaluation":"2024-03-26T23:19:37.013212097Z","type":"alerting"}],"interval":30,"evaluationTime":0.000429041,"lastEvaluation":"2024-03-26T23:19:37.013196392Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103189,"lastEvaluation":"2024-03-26T23:19:36.44844595Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198079,"lastEvaluation":"2024-03-26T23:19:36.448139661Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000103349,"lastEvaluation":"2024-03-26T23:19:36.448340667Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000098127,"lastEvaluation":"2024-03-26T23:19:36.448550856Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000089822,"lastEvaluation":"2024-03-26T23:19:36.448880852Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000151415,"lastEvaluation":"2024-03-26T23:19:36.448727658Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000075685,"lastEvaluation":"2024-03-26T23:19:36.448650515Z","type":"recording"}],"interval":30,"evaluationTime":0.000841111,"lastEvaluation":"2024-03-26T23:19:36.448132532Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000533662,"lastEvaluation":"2024-03-26T23:19:37.069383202Z","type":"recording"}],"interval":30,"evaluationTime":0.000556913,"lastEvaluation":"2024-03-26T23:19:37.06936707Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.002719051,"lastEvaluation":"2024-03-26T23:19:36.022758384Z","type":"alerting"}],"interval":30,"evaluationTime":0.002735718,"lastEvaluation":"2024-03-26T23:19:36.022746231Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"eva{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000322416,"lastEvaluation":"2024-03-26T23:19:37.790631357Z","type":"alerting"}],"interval":30,"evaluationTime":0.000338648,"lastEvaluation":"2024-03-26T23:19:37.790619713Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000181954,"lastEvaluation":"2024-03-26T23:19:40.031858534Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000277207,"lastEvaluation":"2024-03-26T23:19:40.030081605Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000743597,"lastEvaluation":"2024-03-26T23:19:40.030743064Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000380758,"lastEvaluation":"2024-03-26T23:19:40.0303609Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.000368844,"lastEvaluation":"2024-03-26T23:19:40.031488224Z","type":"alerting"}],"interval":30,"evaluationTime":0.001973544,"lastEvaluation":"2024-03-26T23:19:40.030070589Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000356128,"lastEvaluation":"2024-03-26T23:19:37.559085906Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000158335,"lastEvaluation":"2024-03-26T23:19:37.559986611Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000333151,"lastEvaluation":"2024-03-26T23:19:37.558748885Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00053952,"lastEvaluation":"2024-03-26T23:19:37.559444268Z","type":"alerting"}],"interval":30,"evaluationTime":0.001407584,"lastEvaluation":"2024-03-26T23:19:37.558740959Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000249735,"lastEvaluation":"2024-03-26T23:19:38.897747868Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000272005,"lastEvaluation":"2024-03-26T23:19:38.897166207Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000305672,"lastEvaluation":"2024-03-26T23:19:38.897440612Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000301115,"lastEvaluation":"2024-03-26T23:19:38.896862697Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000271221,"lastEvaluation":"2024-03-26T23:19:38.89658998Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.00029068,"lastEvaluation":"2024-03-26T23:19:38.896297931Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000149587,"lastEvaluation":"2024-03-26T23:19:38.896146933Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000149126,"lastEvaluation":"2024-03-26T23:19:38.895996257Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.000548599,"lastEvaluation":"2024-03-26T23:19:38.895311936Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000131843,"lastEvaluation":"2024-03-26T23:19:38.895862615Z","type":"alerting"}],"interval":30,"evaluationTime":0.002699241,"lastEvaluation":"2024-03-26T23:19:38.895301411Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000124789,"lastEvaluation":"2024-03-26T23:19:42.634783154Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000101937,"lastEvaluation":"2024-03-26T23:19:42.634909663Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.000107456,"lastEvaluation":"2024-03-26T23:19:42.634674165Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000107933,"lastEvaluation":"2024-03-26T23:19:42.635013171Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000233467,"lastEvaluation":"2024-03-26T23:19:42.634438926Z","type":"alerting"}],"interval":30,"evaluationTime":0.000695228,"lastEvaluation":"2024-03-26T23:19:42.634428883Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000395125,"lastEvaluation":"2024-03-26T23:20:10.034645897Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000572266,"lastEvaluation":"2024-03-26T23:20:10.030396967Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.00162457,"lastEvaluation":"2024-03-26T23:20:10.031935043Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000956651,"lastEvaluation":"2024-03-26T23:20:10.030974917Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.001077533,"lastEvaluation":"2024-03-26T23:20:10.033563948Z","type":"alerting"}],"interval":30,"evaluationTime":0.004665296,"lastEvaluation":"2024-03-26T23:20:10.030380586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"2024-03-26T23:20:08.899763907Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.001156355,"lastEvaluation":"2024-03-26T23:20:08.898603972Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000748404,"lastEvaluation":"2024-03-26T23:20:08.897850749Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000380934,"lastEvaluation":"2024-03-26T23:20:08.897455708Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000347778,"lastEvaluation":"2024-03-26T23:20:08.897103847Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.001301448,"lastEvaluation":"2024-03-26T23:20:08.895511424Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000275968,"lastEvaluation":"2024-03-26T23:20:08.896819204Z","type":"alerting"}],"interval":30,"evaluationTime":0.006396776,"lastEvaluation":"2024-03-26T23:20:08.895492559Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000392275,"lastEvaluation":"2024-03-26T23:19:41.819466055Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000729582,"lastEvaluation":"2024-03-26T23:19:41.818731639Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000297709,"lastEvaluation":"2024-03-26T23:19:41.819862233Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000290967,"lastEvaluation":"2024-03-26T23:19:41.820162375Z","type":"recording"}],"interval":30,"evaluationTime":0.001745721,"lastEvaluation":"2024-03-26T23:19:41.818711315Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000126728,"lastEvaluation":"2024-03-26T23:19:41.221108035Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000336207,"lastEvaluation":"2024-03-26T23:19:41.220178603Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.00038708,"lastEvaluation":"2024-03-26T23:19:41.220518374Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000198687,"lastEvaluation":"2024-03-26T23:19:41.220907411Z","type":"alerting"}],"interval":30,"evaluationTime":0.001072506,"lastEvaluation":"2024-03-26T23:19:41.220167235Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000341416,"lastEvaluation":"2024-03-26T23:19:42.465007251Z","type":"alerting"}],"interval":30,"evaluationTime":0.000354629,"lastEvaluation":"2024-03-26T23:19:42.464999974Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000139903,"lastEvaluation":"2024-03-26T23:20:12.635267096Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000164241,"lastEvaluation":"2024-03-26T23:20:12.63540953Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.00019769,"lastEvaluation":"2024-03-26T23:20:12.635061792Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000158249,"lastEvaluation":"2024-03-26T23:20:12.635576586Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000496007,"lastEvaluation":"2024-03-26T23:20:12.634561549Z","type":"alerting"}],"interval":30,"evaluationTime":0.001189226,"lastEvaluation":"2024-03-26T23:20:12.63455098Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000156684,"lastEvaluation":"2024-03-26T23:19:44.799027706Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000563575,"lastEvaluation":"2024-03-26T23:19:44.7984616Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000157322,"lastEvaluation":"2024-03-26T23:19:44.799187221Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.00016648,"lastEvaluation":"2024-03-26T23:19:44.799347991Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.000156662,"lastEvaluation":"2024-03-26T23:19:44.799516984Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000776779,"lastEvaluation":"2024-03-26T23:19:44.797681715Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000132787,"lastEvaluation":"2024-03-26T23:19:44.799675915Z","type":"alerting"}],"interval":30,"evaluationTime":0.002139232,"lastEvaluation":"2024-03-26T23:19:44.797672452Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00041666,"lastEvaluation":"2024-03-26T23:19:45.725574753Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108369,"lastEvaluation":"2024-03-26T23:19:45.72612994Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000134288,"lastEvaluation":"2024-03-26T23:19:45.725993919Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000108472,"lastEvaluation":"2024-03-26T23:19:45.726240101Z","type":"recording"}],"interval":30,"evaluationTime":0.000794671,"lastEvaluation":"2024-03-26T23:19:45.72555723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000395125,"lastEvaluation":"2024-03-26T23:20:10.034645897Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000572266,"lastEvaluation":"2024-03-26T23:20:10.030396967Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.00162457,"lastEvaluation":"2024-03-26T23:20:10.031935043Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000956651,"lastEvaluation":"2024-03-26T23:20:10.030974917Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.001077533,"lastEvaluation":"2024-03-26T23:20:10.033563948Z","type":"alerting"}],"interval":30,"evaluationTime":0.004665296,"lastEvaluation":"2024-03-26T23:20:10.030380586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"2024-03-26T23:20:08.899763907Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.001156355,"lastEvaluation":"2024-03-26T23:20:08.898603972Z","type":"alerting"},{"state":"inactive","name":"etcdHighFsyncDurations","query":"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.5","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile fsync durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000748404,"lastEvaluation":"2024-03-26T23:20:08.897850749Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedProposals","query":"rate(etcd_server_proposals_failed_total{job=~\".*etcd.*\"}[15m]) \u003e 5","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of proposal failures."},"alerts":[],"health":"ok","evaluationTime":0.000380934,"lastEvaluation":"2024-03-26T23:20:08.897455708Z","type":"alerting"},{"state":"inactive","name":"etcdMemberCommunicationSlow","query":"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.15","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster member communication is slow."},"alerts":[],"health":"ok","evaluationTime":0.000347778,"lastEvaluation":"2024-03-26T23:20:08.897103847Z","type":"alerting"},{"state":"inactive","name":"etcdMembersDown","query":"max without (endpoint) (sum without (instance) (up{job=~\".*etcd.*\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\".*etcd.*\"}[2m])) \u003e 0.01)) \u003e 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": members are down ({{ $value }}).","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md","summary":"etcd cluster members are down."},"alerts":[],"health":"ok","evaluationTime":0.001301448,"lastEvaluation":"2024-03-26T23:20:08.895511424Z","type":"alerting"},{"state":"inactive","name":"etcdNoLeader","query":"etcd_server_has_leader{job=~\".*etcd.*\"} == 0","duration":60,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": member {{ $labels.instance }} has no leader.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md","summary":"etcd cluster has no leader."},"alerts":[],"health":"ok","evaluationTime":0.000275968,"lastEvaluation":"2024-03-26T23:20:08.896819204Z","type":"alerting"}],"interval":30,"evaluationTime":0.006396776,"lastEvaluation":"2024-03-26T23:20:08.895492559Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-etcd.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdGRPCRequestsSlow","query":"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\"Defragment\",grpc_type=\"unary\",job=\"etcd\"}[10m]))) \u003e 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md","summary":"etcd grpc requests are slow"},"alerts":[],"health":"ok","evaluationTime":0.013340445,"lastEvaluation":"2024-03-26T23:20:01.396035613Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 50","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.022414878,"lastEvaluation":"2024-03-26T23:20:01.430286042Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfFailedGRPCRequests","query":"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\",job=\"etcd\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\"etcd\"}[5m])) \u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\"ipi|BareMetal\"} == bool 1)))) * 100 \u003e 10","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.","summary":"etcd cluster has high number of failed grpc requests."},"alerts":[],"health":"ok","evaluationTime":0.020897431,"lastEvaluation":"2024-03-26T23:20:01.409382006Z","type":"alerting"},{"state":"inactive","name":"etcdHighNumberOfLeaderChanges","query":"avg(changes(etcd_server_is_leader[10m])) \u003e 5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.","summary":"etcd cluster has high number of leader changes."},"alerts":[],"health":"ok","evaluationTime":0.000387678,"lastEvaluation":"2024-03-26T23:20:01.45270739Z","type":"alerting"},{"state":"inactive","name":"etcdInsufficientMembers","query":"sum without (instance, pod) (up{job=\"etcd\"} == bool 1 and etcd_server_has_leader{job=\"etcd\"} == bool 1) \u003c ((count without (instance, pod) (up{job=\"etcd\"}) + 1) / 2)","duration":180,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md","summary":"etcd is reporting that a majority of instances are unavailable."},"alerts":[],"health":"ok","evaluationTime":0.000717035,"lastEvaluation":"2024-03-26T23:20:01.453098291Z","type":"alerting"}],"interval":30,"evaluationTime":0.057795073,"lastEvaluation":"2024-03-26T23:20:01.396024616Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imageregistry.operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-6d07ab8c-fe87-472d-99be-2c2170baccce.yaml","rules":[{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.Create\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000573366,"lastEvaluation":"2024-03-26T23:20:11.819467845Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"BlobStore.ServeBlob\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"blob\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000760536,"lastEvaluation":"2024-03-26T23:20:11.818702105Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Get\"}), \"operation\", \"get\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000514415,"lastEvaluation":"2024-03-26T23:20:11.820045564Z","type":"recording"},{"name":"imageregistry:operations_count:sum","query":"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\"ManifestService.Put\"}), \"operation\", \"create\", \"operation\", \"(.+)\"), \"resource_type\", \"manifest\", \"resource_type\", \"\")","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000457601,"lastEvaluation":"2024-03-26T23:20:11.820565464Z","type":"recording"}],"interval":30,"evaluationTime":0.002342689,"lastEvaluation":"2024-03-26T23:20:11.818685752Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"imagestreams.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-11808d4b-c62d-45bf-89e7-001ed41db836.yaml","rules":[{"name":"imageregistry:imagestreamtags_count:sum","query":"sum by (location, source) (image_registry_image_stream_tags_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000680337,"lastEvaluation":"2024-03-26T23:20:07.068586362Z","type":"recording"}],"interval":30,"evaluationTime":0.000712893,"lastEvaluation":"2024-03-26T23:20:07.068560702Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress-to-route-controller.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"IngressWithoutClassName","query":"openshift_ingress_to_route_controller_ingress_without_class_name == 1","duration":86400,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.","message":"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.","summary":"Ingress without IngressClassName for 1 day"},"alerts":[],"health":"ok","evaluationTime":0.000527187,"lastEvaluation":"2024-03-26T23:20:03.670207806Z","type":"alerting"},{"state":"inactive","name":"UnmanagedRoutes","query":"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when there is a Route owned by an unmanaged Ingress.","message":"Route {{ $labels.name }} is owned by an unmanaged Ingress.","summary":"Route owned by an Ingress no longer managed"},"alerts":[],"health":"ok","evaluationTime":0.001873896,"lastEvaluation":"2024-03-26T23:20:03.670740305Z","type":"alerting"}],"interval":30,"evaluationTime":0.00243225,"lastEvaluation":"2024-03-26T23:20:03.670187701Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-ingress.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-98b8c560-d94b-43ff-b8ba-3189afcf28bd.yaml","rules":[{"state":"inactive","name":"HAProxyDown","query":"haproxy_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"This alert fires when metrics report that HAProxy is down.","message":"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}","summary":"HAProxy is down"},"alerts":[],"health":"ok","evaluationTime":0.00018611,"lastEvaluation":"2024-03-26T23:19:56.886372167Z","type":"alerting"},{"state":"inactive","name":"HAProxyReloadFail","query":"template_router_reload_failure == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.","message":"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes","summary":"HAProxy reload failure"},"alerts":[],"health":"ok","evaluationTime":0.000444696,"lastEvaluation":"2024-03-26T23:19:56.885922365Z","type":"alerting"},{"state":"inactive","name":"IngressControllerDegraded","query":"ingress_controller_conditions{condition=\"Degraded\"} == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController status is degraded.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\ndegraded: {{ $labels.reason }}.\n","summary":"IngressController is degraded"},"alerts":[],"health":"ok","evaluationTime":0.000239711,"lastEvaluation":"2024-03-26T23:19:56.886563621Z","type":"alerting"},{"state":"inactive","name":"IngressControllerUnavailable","query":"ingress_controller_conditions{condition=\"Available\"} == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"This alert fires when the IngressController is not available.","message":"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\nunavailable: {{ $labels.reason }}.\n","summary":"IngressController is unavailable"},"alerts":[],"health":"ok","evaluationTime":0.000217637,"lastEvaluation":"2024-03-26T23:19:56.886807571Z","type":"alerting"},{"name":"cluster:openshift_route_info:tls_termination:sum","query":"sum by (tls_termination) (openshift_route_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000561048,"lastEvaluation":"2024-03-26T23:19:56.887885874Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:avg","query":"avg(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000166798,"lastEvaluation":"2024-03-26T23:19:56.887513175Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:max","query":"max(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.0001727,"lastEvaluation":"2024-03-26T23:19:56.887336861Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:median","query":"quantile(0.5, route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000198515,"lastEvaluation":"2024-03-26T23:19:56.887684055Z","type":"recording"},{"name":"cluster:route_metrics_controller_routes_per_shard:min","query":"min(route_metrics_controller_routes_per_shard)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303696,"lastEvaluation":"2024-03-26T23:19:56.887029101Z","type":"recording"}],"interval":30,"evaluationTime":0.002551443,"lastEvaluation":"2024-03-26T23:19:56.885899863Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"insights","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-c2d9cb8e-c45e-4ef0-8ac5-14639b46c4d4.yaml","rules":[{"state":"inactive","name":"InsightsDisabled","query":"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\"Disabled\",name=\"insights\"} == 1)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html","summary":"Insights operator is disabled."},"alerts":[],"health":"ok","evaluationTime":0.000589563,"lastEvaluation":"2024-03-26T23:19:44.624408867Z","type":"alerting"},{"state":"firing","name":"InsightsRecommendationActive","query":"insights_recommendation_active == 1","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Insights recommendation \"{{ $labels.description }}\" with total risk \"{{ $labels.total_risk }}\" was detected on the cluster. More information is available at {{ $labels.info_link }}.","summary":"An Insights recommendation is active for this cluster."},"alerts":[{"labels":{"alertname":"InsightsRecommendationActive","container":"insights-operator","description":"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them","endpoint":"https","info_link":"https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION","instance":"10.128.0.52:8443","job":"metrics","namespace":"openshift-insights","pod":"insights-operator-55df4f9fcc-d6p95","service":"metrics","severity":"info","total_risk":"Important"},"annotations":{"description":"Insights recommendation \"SystemMemoryExceedsReservation alerts when the system daemons memory usage on nodes exceeds 90% of the reservation for them\" with total risk \"Important\" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/d0fcf4b2-ed6c-49f1-8677-d0b646c69971?first=ccx_rules_ocp.external.rules.system_memory_exceed_reservation|SYSTEM_MEMORY_EXCEEDS_RESERVATION.","summary":"An Insights recommendation is active for this cluster."},"state":"firing","activeAt":"2024-03-26T21:54:44.622850178Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.001305178,"lastEvaluation":"2024-03-26T23:19:44.625902048Z","type":"alerting"},{"state":"firing","name":"SimpleContentAccessNotAvailable","query":"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\"SCAAvailable\",name=\"insights\",reason=\"NotFound\"}[5m]) == 0)","duration":300,"labels":{"namespace":"openshift-insights","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"alerts":[{"labels":{"alertname":"SimpleContentAccessNotAvailable","condition":"SCAAvailable","endpoint":"metrics","name":"insights","namespace":"openshift-insights","reason":"NotFound","severity":"info"},"annotations":{"description":"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.","summary":"Simple content access certificates are not available."},"state":"firing","activeAt":"2024-03-26T21:11:44.622850178Z","value":"0e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000894442,"lastEvaluation":"2024-03-26T23:19:44.625003717Z","type":"alerting"}],"interval":30,"evaluationTime":0.002823043,"lastEvaluation":"2024-03-26T23:19:44.624388517Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pre-release-lifecycle","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-6aa67305-a8d8-4ea9-ae6a-187677151fc0.yaml","rules":[{"state":"inactive","name":"APIRemovedInNextEUSReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next EUS version is being used."},"alerts":[],"health":"ok","evaluationTime":0.118429892,"lastEvaluation":"2024-03-26T23:19:58.124778148Z","type":"alerting"},{"state":"inactive","name":"APIRemovedInNextReleaseInUse","query":"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\"1.29\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\"cluster-policy-controller\",system_client!=\"kube-controller-manager\"}[4h])) \u003e 0","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.","summary":"Deprecated API that will be removed in the next version is being used."},"alerts":[],"health":"ok","evaluationTime":0.111678536,"lastEvaluation":"2024-03-26T23:19:58.013088321Z","type":"alerting"}],"interval":30,"evaluationTime":0.230145948,"lastEvaluation":"2024-03-26T23:19:58.013070565Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-audit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-c7069c98-28fb-4569-ac52-53aa32583fff.yaml","rules":[{"state":"inactive","name":"AuditLogError","query":"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\".+-apiserver\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\".+-apiserver\"}[5m])) \u003e 0","duration":60,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"An API Server had an error writing to an audit log.","summary":"An API Server instance was unable to write audit logs. This could be\ntriggered by the node running out of space, or a malicious actor\ntampering with the audit logs."},"alerts":[],"health":"ok","evaluationTime":0.000663169,"lastEvaluation":"2024-03-26T23:19:43.508361631Z","type":"alerting"}],"interval":30,"evaluationTime":0.000680315,"lastEvaluation":"2024-03-26T23:19:43.508349664Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"control-plane-cpu-utilization","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-0e48fa97-2ba9-4be3-a3d2-b98061352355.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":3600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000802609,"lastEvaluation":"2024-03-26T23:19:55.927566401Z","type":"alerting"},{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneCPU","query":"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) \u003e 90 and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")","duration":300,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.000775373,"lastEvaluation":"2024-03-26T23:19:55.926787823Z","type":"alerting"},{"state":"inactive","name":"HighOverallControlPlaneCPU","query":"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / count(kube_node_role{role=\"master\"}) \u003e 60","duration":600,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md","summary":"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU."},"alerts":[],"health":"ok","evaluationTime":0.001190874,"lastEvaluation":"2024-03-26T23:19:55.92559363Z","type":"alerting"}],"interval":30,"evaluationTime":0.002787579,"lastEvaluation":"2024-03-26T23:19:55.925586291Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"api-performance","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-8e9ff98d-6e24-4080-994f-d189a7e732d7.yaml","rules":[{"name":"cluster:apiserver_tls_handshake_errors_total:rate1m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000426719,"lastEvaluation":"2024-03-26T23:19:55.968719415Z","type":"recording"},{"name":"cluster:apiserver_tls_handshake_errors_total:rate5m","query":"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000359261,"lastEvaluation":"2024-03-26T23:19:55.969149272Z","type":"recording"},{"name":"code:apiserver_request_total:rate1m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.053119382,"lastEvaluation":"2024-03-26T23:19:55.628409063Z","type":"recording"},{"name":"code:apiserver_request_total:rate5m","query":"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.050824022,"lastEvaluation":"2024-03-26T23:19:55.681537045Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:1m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000827687,"lastEvaluation":"2024-03-26T23:19:55.626655981Z","type":"recording"},{"name":"component_resource:apiserver_request_terminations_total:rate:5m","query":"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000919153,"lastEvaluation":"2024-03-26T23:19:55.627486706Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate1m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005119788,"lastEvaluation":"2024-03-26T23:19:55.930328263Z","type":"recording"},{"name":"filter:apiserver_request_filter_duration_seconds_bucket:rate5m","query":"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00382836,"lastEvaluation":"2024-03-26T23:19:55.935453491Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.001460595,"lastEvaluation":"2024-03-26T23:19:55.918360619Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum","query":"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000701218,"lastEvaluation":"2024-03-26T23:19:55.920594771Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.008319428,"lastEvaluation":"2024-03-26T23:19:55.9009009Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00912013,"lastEvaluation":"2024-03-26T23:19:55.909228222Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006957511,"lastEvaluation":"2024-03-26T23:19:55.85334411Z","type":"recording"},{"name":"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",execute=\"true\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006258872,"lastEvaluation":"2024-03-26T23:19:55.860306971Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.017416219,"lastEvaluation":"2024-03-26T23:19:55.867196343Z","type":"recording"},{"name":"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016265582,"lastEvaluation":"2024-03-26T23:19:55.884625299Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000302613,"lastEvaluation":"2024-03-26T23:19:55.866569864Z","type":"recording"},{"name":"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000317092,"lastEvaluation":"2024-03-26T23:19:55.866876369Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003495887,"lastEvaluation":"2024-03-26T23:19:55.846367869Z","type":"recording"},{"name":"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m","query":"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003474387,"lastEvaluation":"2024-03-26T23:19:55.849866362Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00607514,"lastEvaluation":"2024-03-26T23:19:55.949983517Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_sizes_sum:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005673356,"lastEvaluation":"2024-03-26T23:19:55.95606483Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate1m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00437515,"lastEvaluation":"2024-03-26T23:19:55.939286028Z","type":"recording"},{"name":"group_kind:apiserver_watch_events_total:rate5m","query":"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006311526,"lastEvaluation":"2024-03-26T23:19:55.943667167Z","type":"recording"},{"name":"group_resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006970114,"lastEvaluation":"2024-03-26T23:19:55.961743551Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate1m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000247032,"lastEvaluation":"2024-03-26T23:19:55.626132304Z","type":"recording"},{"name":"group_resource:apiserver_request_total:rate5m","query":"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",code=\"429\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000270362,"lastEvaluation":"2024-03-26T23:19:55.626382611Z","type":"recording"},{"name":"instance:apiserver_longrunning_requests:sum","query":"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.002244017,"lastEvaluation":"2024-03-26T23:19:55.806854399Z","type":"recording"},{"name":"instance:apiserver_request_total:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.044499115,"lastEvaluation":"2024-03-26T23:19:55.732370334Z","type":"recording"},{"name":"instance:apiserver_request_total:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.026496207,"lastEvaluation":"2024-03-26T23:19:55.776882486Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.006694411,"lastEvaluation":"2024-03-26T23:19:55.809342516Z","type":"recording"},{"name":"instance:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005723282,"lastEvaluation":"2024-03-26T23:19:55.816039298Z","type":"recording"},{"name":"instance_request_kind:apiserver_current_inflight_requests:sum","query":"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000238987,"lastEvaluation":"2024-03-26T23:19:55.809101762Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.105819716,"lastEvaluation":"2024-03-26T23:19:53.889027344Z","type":"recording"},{"name":"list:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.144476689,"lastEvaluation":"2024-03-26T23:19:53.994859472Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate1m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.2638198,"lastEvaluation":"2024-03-26T23:19:54.877384629Z","type":"recording"},{"name":"operation:etcd_request_duration_seconds_bucket:rate5m","query":"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.332886896,"lastEvaluation":"2024-03-26T23:19:55.141213751Z","type":"recording"},{"name":"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum","query":"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000762182,"lastEvaluation":"2024-03-26T23:19:55.919828672Z","type":"recording"},{"name":"read:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.014589921,"lastEvaluation":"2024-03-26T23:19:55.561379219Z","type":"recording"},{"name":"read:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.020373529,"lastEvaluation":"2024-03-26T23:19:55.575971487Z","type":"recording"},{"name":"resource:apiserver_longrunning_requests:sum","query":"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003462856,"lastEvaluation":"2024-03-26T23:19:55.803388773Z","type":"recording"},{"name":"resource:apiserver_storage_objects:max","query":"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.005742609,"lastEvaluation":"2024-03-26T23:19:55.969511647Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000709902,"lastEvaluation":"2024-03-26T23:19:55.928991065Z","type":"recording"},{"name":"resource_verb:apiserver_request_aborts_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000620082,"lastEvaluation":"2024-03-26T23:19:55.929704827Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.426968511,"lastEvaluation":"2024-03-26T23:19:53.217422618Z","type":"recording"},{"name":"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m","query":"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.244604411,"lastEvaluation":"2024-03-26T23:19:53.644403962Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.038769368,"lastEvaluation":"2024-03-26T23:19:55.474116195Z","type":"recording"},{"name":"resource_verb:apiserver_request_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.048471191,"lastEvaluation":"2024-03-26T23:19:55.512897364Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013017224,"lastEvaluation":"2024-03-26T23:19:55.821766185Z","type":"recording"},{"name":"resource_verb:apiserver_response_sizes_sum:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.011565664,"lastEvaluation":"2024-03-26T23:19:55.834793135Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate1m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.004228049,"lastEvaluation":"2024-03-26T23:19:55.921299523Z","type":"recording"},{"name":"resource_verb:apiserver_selfrequest_total:rate5m","query":"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.003450263,"lastEvaluation":"2024-03-26T23:19:55.92553668Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.19864672,"lastEvaluation":"2024-03-26T23:19:54.430265988Z","type":"recording"},{"name":"verb:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb!~\"WATCH|WATCHLIST|PROXY\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.248446775,"lastEvaluation":"2024-03-26T23:19:54.628921284Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate1m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.1531449,"lastEvaluation":"2024-03-26T23:19:54.139352683Z","type":"recording"},{"name":"write:apiserver_request_duration_seconds_bucket:rate5m","query":"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.137744343,"lastEvaluation":"2024-03-26T23:19:54.292515701Z","type":"recording"},{"name":"write:apiserver_request_total:rate1m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[1m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.013754106,"lastEvaluation":"2024-03-26T23:19:55.596350341Z","type":"recording"},{"name":"write:apiserver_request_total:rate5m","query":"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\",verb=~\"POST|PUT|PATCH|UPDATE|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.016022553,"lastEvaluation":"2024-03-26T23:19:55.610107301Z","type":"recording"}],"interval":30,"evaluationTime":2.757862094,"lastEvaluation":"2024-03-26T23:19:53.217399151Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"apiserver-requests-in-flight","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-47c7d7c6-96e2-4f6b-bd2e-ad52d624cb11.yaml","rules":[{"name":"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m","query":"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\"openshift-apiserver|kube-apiserver\"})[2m:])","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000494577,"lastEvaluation":"2024-03-26T23:19:49.574672085Z","type":"recording"}],"interval":30,"evaluationTime":0.000511482,"lastEvaluation":"2024-03-26T23:19:49.574658675Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver-slos-basic","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate1h) \u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \u003e (14.4 * 0.01)","duration":120,"labels":{"long":"1h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"5m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000243476,"lastEvaluation":"2024-03-26T23:19:48.062687701Z","type":"alerting"},{"state":"inactive","name":"KubeAPIErrorBudgetBurn","query":"sum(apiserver_request:burnrate6h) \u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \u003e (6 * 0.01)","duration":900,"labels":{"long":"6h","namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"critical","short":"30m"},"annotations":{"description":"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md","summary":"The API server is burning too much error budget."},"alerts":[],"health":"ok","evaluationTime":0.000185802,"lastEvaluation":"2024-03-26T23:19:48.062933Z","type":"alerting"}],"interval":30,"evaluationTime":0.000438654,"lastEvaluation":"2024-03-26T23:19:48.062682273Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"kube-apiserver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-3ea04165-a643-4582-80b4-79e737b2047a.yaml","rules":[{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.019127789,"lastEvaluation":"2024-03-26T23:19:50.760676408Z","type":"recording"},{"name":"apiserver_request:burnrate1h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[1h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.01721134,"lastEvaluation":"2024-03-26T23:19:50.820851213Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.016218126,"lastEvaluation":"2024-03-26T23:19:50.744456008Z","type":"recording"},{"name":"apiserver_request:burnrate30m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[30m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.014244401,"lastEvaluation":"2024-03-26T23:19:50.838065727Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.011139043,"lastEvaluation":"2024-03-26T23:19:50.733314282Z","type":"recording"},{"name":"apiserver_request:burnrate5m","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.007522909,"lastEvaluation":"2024-03-26T23:19:50.852313801Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",scope=~\"resource|\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"5\",scope=\"namespace\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"30\",scope=\"cluster\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"LIST|GET\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.041035471,"lastEvaluation":"2024-03-26T23:19:50.779806989Z","type":"recording"},{"name":"apiserver_request:burnrate6h","query":"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\"apiserver\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\"apiserver\",le=\"1\",subresource!~\"proxy|attach|log|exec|portforward\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\"5..\",job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[6h]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.022879927,"lastEvaluation":"2024-03-26T23:19:50.85983984Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.5"},"health":"ok","evaluationTime":0.156940519,"lastEvaluation":"2024-03-26T23:19:51.40546962Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.9"},"health":"ok","evaluationTime":0.162963187,"lastEvaluation":"2024-03-26T23:19:51.242497362Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",subresource!=\"log\",verb!~\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\"}[5m])))","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99"},"health":"ok","evaluationTime":0.175428251,"lastEvaluation":"2024-03-26T23:19:51.067060184Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"read"},"health":"ok","evaluationTime":0.077958856,"lastEvaluation":"2024-03-26T23:19:50.906963626Z","type":"recording"},{"name":"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile","query":"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))) \u003e 0","labels":{"prometheus":"openshift-monitoring/k8s","quantile":"0.99","verb":"write"},"health":"ok","evaluationTime":0.082127439,"lastEvaluation":"2024-03-26T23:19:50.984926629Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"LIST|GET\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"read"},"health":"ok","evaluationTime":0.01190833,"lastEvaluation":"2024-03-26T23:19:50.882726954Z","type":"recording"},{"name":"code_resource:apiserver_request_total:rate5m","query":"sum by (code, resource) (rate(apiserver_request_total{job=\"apiserver\",verb=~\"POST|PUT|PATCH|DELETE\"}[5m]))","labels":{"prometheus":"openshift-monitoring/k8s","verb":"write"},"health":"ok","evaluationTime":0.012317501,"lastEvaluation":"2024-03-26T23:19:50.894639815Z","type":"recording"}],"interval":30,"evaluationTime":0.829104675,"lastEvaluation":"2024-03-26T23:19:50.733312155Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-4a2e8a18-0bc4-4c80-9e45-69715eebd6f1.yaml","rules":[{"state":"inactive","name":"TechPreviewNoUpgrade","query":"cluster_feature_set{name!=\"\",namespace=\"openshift-kube-apiserver-operator\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.","summary":"Cluster has enabled tech preview features that will prevent upgrades."},"alerts":[],"health":"ok","evaluationTime":0.005640666,"lastEvaluation":"2024-03-26T23:20:06.022768547Z","type":"alerting"}],"interval":30,"evaluationTime":0.005667944,"lastEvaluation":"2024-03-26T23:20:06.022749252Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"pod-security-violation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-a5be1d15-de3e-4845-9798-b232bce86921.yaml","rules":[{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace!=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created in namespace \"{{ $labels.ocp_namespace }}\" but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000363553,"lastEvaluation":"2024-03-26T23:20:03.526656532Z","type":"alerting"},{"state":"inactive","name":"PodSecurityViolation","query":"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\"deny\",mode=\"audit\",ocp_namespace=\"\",resource=\"pod\"}[1d])) \u003e 0","duration":0,"labels":{"namespace":"openshift-kube-apiserver","prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \"{{ $labels.policy_level }}\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \"restricted\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.","summary":"One or more workloads users created in the cluster don't match their Pod Security profile"},"alerts":[],"health":"ok","evaluationTime":0.000838788,"lastEvaluation":"2024-03-26T23:20:03.525812052Z","type":"alerting"}],"interval":30,"evaluationTime":0.001225321,"lastEvaluation":"2024-03-26T23:20:03.525799845Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-671153c7-cd1a-4135-93bd-2ca5dbf6a70b.yaml","rules":[{"state":"inactive","name":"GarbageCollectorSyncFailed","query":"rate(garbagecollector_controller_resources_sync_error_total[5m]) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md","summary":"There was a problem with syncing the resources for garbage collection."},"alerts":[],"health":"ok","evaluationTime":0.000314126,"lastEvaluation":"2024-03-26T23:20:11.222541907Z","type":"alerting"},{"state":"inactive","name":"KubeControllerManagerDown","query":"absent(up{job=\"kube-controller-manager\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-controller-manager","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeControllerManager has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.00051606,"lastEvaluation":"2024-03-26T23:20:11.220868186Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetAtLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \u003e 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md","summary":"The pod disruption budget is preventing further disruption to pods."},"alerts":[],"health":"ok","evaluationTime":0.000718783,"lastEvaluation":"2024-03-26T23:20:11.221389626Z","type":"alerting"},{"state":"inactive","name":"PodDisruptionBudgetLimit","query":"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \u003c kube_poddisruptionbudget_status_desired_healthy)","duration":900,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md","summary":"The pod disruption budget registers insufficient amount of pods."},"alerts":[],"health":"ok","evaluationTime":0.000425865,"lastEvaluation":"2024-03-26T23:20:11.222111886Z","type":"alerting"}],"interval":30,"evaluationTime":0.002009829,"lastEvaluation":"2024-03-26T23:20:11.220851723Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"KubeSchedulerDown","query":"absent(up{job=\"scheduler\"} == 1)","duration":900,"labels":{"namespace":"openshift-kube-scheduler","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"KubeScheduler has disappeared from Prometheus target discovery.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md","summary":"Target disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000409578,"lastEvaluation":"2024-03-26T23:19:47.024400593Z","type":"alerting"}],"interval":30,"evaluationTime":0.000428047,"lastEvaluation":"2024-03-26T23:19:47.024386698Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"scheduler-legacy-policy-deprecated","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-8dab3368-3211-4d6e-a798-3d9b60656ae0.yaml","rules":[{"state":"inactive","name":"SchedulerLegacyPolicySet","query":"cluster_legacy_scheduler_policy \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.","summary":"Legacy scheduler policy API in use by the scheduler."},"alerts":[],"health":"ok","evaluationTime":0.000365438,"lastEvaluation":"2024-03-26T23:19:52.956202166Z","type":"alerting"}],"interval":30,"evaluationTime":0.000391995,"lastEvaluation":"2024-03-26T23:19:52.956186567Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-api-operator-metrics-collector-up","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineAPIOperatorMetricsCollectionFailing","query":"mapi_mao_collector_up == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"For more details:  oc logs \u003cmachine-api-operator-pod-name\u003e -n openshift-machine-api","summary":"machine api operator metrics collection is failing."},"alerts":[],"health":"ok","evaluationTime":0.000491507,"lastEvaluation":"2024-03-26T23:19:44.304344609Z","type":"alerting"}],"interval":30,"evaluationTime":0.000518527,"lastEvaluation":"2024-03-26T23:19:44.304323543Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-health-check-unterminated-short-circuit","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineHealthCheckUnterminatedShortCircuit","query":"mapi_machinehealthcheck_short_circuit == 1","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n","summary":"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"},"alerts":[],"health":"ok","evaluationTime":0.000239276,"lastEvaluation":"2024-03-26T23:19:49.733712826Z","type":"alerting"}],"interval":30,"evaluationTime":0.000259522,"lastEvaluation":"2024-03-26T23:19:49.733698623Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-not-yet-deleted","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineNotYetDeleted","query":"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\"Deleting\"}[15m])) \u003e 0","duration":21600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine is not properly deleting, this may be due to a configuration issue with the\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\nlong termination periods which are preventing deletion.\n","summary":"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours"},"alerts":[],"health":"ok","evaluationTime":0.000568622,"lastEvaluation":"2024-03-26T23:20:12.465329754Z","type":"alerting"}],"interval":30,"evaluationTime":0.000584969,"lastEvaluation":"2024-03-26T23:20:12.465318631Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-with-no-running-phase","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithNoRunningPhase","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\"Running|Deleting\"}) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The machine has been without a Running or Deleting phase for more than 60 minutes.\nThe machine may not have been provisioned properly from the infrastructure provider, or\nit might have issues with CertificateSigningRequests being approved.\n","summary":"machine {{ $labels.name }} is in phase: {{ $labels.phase }}"},"alerts":[],"health":"ok","evaluationTime":0.000786854,"lastEvaluation":"2024-03-26T23:20:00.563512905Z","type":"alerting"}],"interval":30,"evaluationTime":0.000812595,"lastEvaluation":"2024-03-26T23:20:00.563494815Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"machine-without-valid-node-ref","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-658e576c-6bc2-45cf-ab99-c2150bd052c3.yaml","rules":[{"state":"inactive","name":"MachineWithoutValidNode","query":"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \u003e 0","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"If the machine never became a node, you should diagnose the machine related failures.\nIf the node was deleted from the API, you may delete the machine if appropriate.\n","summary":"machine {{ $labels.name }} does not have valid node reference"},"alerts":[],"health":"ok","evaluationTime":0.000754463,"lastEvaluation":"2024-03-26T23:19:56.39608856Z","type":"alerting"}],"interval":30,"evaluationTime":0.000780961,"lastEvaluation":"2024-03-26T23:19:56.396069777Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcc-drain-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"state":"inactive","name":"MCCDrainError","query":"mcc_drain_err \u003e 0","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller","summary":"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times."},"alerts":[],"health":"ok","evaluationTime":0.000527222,"lastEvaluation":"2024-03-26T23:19:50.406498096Z","type":"alerting"}],"interval":30,"evaluationTime":0.000552065,"lastEvaluation":"2024-03-26T23:19:50.406479966Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcc-pool-alert","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"state":"inactive","name":"MCCPoolAlert","query":"mcc_pool_alert \u003e 0","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller","summary":"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored."},"alerts":[],"health":"ok","evaluationTime":0.000187879,"lastEvaluation":"2024-03-26T23:19:46.287841599Z","type":"alerting"}],"interval":30,"evaluationTime":0.00019628,"lastEvaluation":"2024-03-26T23:19:46.287835828Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"os-image-override.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-fb1acebd-fc1d-42da-a845-f4ed915d82c8.yaml","rules":[{"name":"os_image_url_override:sum","query":"sum(os_image_url_override)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000383984,"lastEvaluation":"2024-03-26T23:19:44.314511353Z","type":"recording"}],"interval":30,"evaluationTime":0.000395439,"lastEvaluation":"2024-03-26T23:19:44.31450672Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"extremely-high-individual-control-plane-memory","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"ExtremelyHighIndividualControlPlaneMemory","query":"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\"))) * 100 \u003e 90","duration":2700,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.","summary":"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability."},"alerts":[],"health":"ok","evaluationTime":0.001066181,"lastEvaluation":"2024-03-26T23:20:12.932909511Z","type":"alerting"}],"interval":30,"evaluationTime":0.00108443,"lastEvaluation":"2024-03-26T23:20:12.932898069Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"high-overall-control-plane-memory","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"HighOverallControlPlaneMemory","query":"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\"master\"}, \"instance\", \"$1\", \"node\", \"(.+)\"))) * 100 \u003e 60","duration":3600,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.","summary":"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability."},"alerts":[],"health":"ok","evaluationTime":0.001730014,"lastEvaluation":"2024-03-26T23:20:10.598304141Z","type":"alerting"}],"interval":30,"evaluationTime":0.001753871,"lastEvaluation":"2024-03-26T23:20:10.598288009Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-kubelet-health-state-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"KubeletHealthState","query":"mcd_kubelet_state \u003e 2","duration":0,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Kubelet health failure threshold reached","summary":"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur."},"alerts":[],"health":"ok","evaluationTime":0.000492102,"lastEvaluation":"2024-03-26T23:20:07.418108354Z","type":"alerting"}],"interval":30,"evaluationTime":0.000515962,"lastEvaluation":"2024-03-26T23:20:07.418090455Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-pivot-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"MCDPivotError","query":"mcd_pivot_errors_total \u003e 0","duration":120,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon ","summary":"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000518095,"lastEvaluation":"2024-03-26T23:20:10.502164905Z","type":"alerting"}],"interval":30,"evaluationTime":0.000546917,"lastEvaluation":"2024-03-26T23:20:10.502147201Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"mcd-reboot-error","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"inactive","name":"MCDRebootError","query":"mcd_reboots_failed_total \u003e 0","duration":300,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon ","summary":"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000299106,"lastEvaluation":"2024-03-26T23:19:47.605849339Z","type":"alerting"}],"interval":30,"evaluationTime":0.000319047,"lastEvaluation":"2024-03-26T23:19:47.605834017Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"system-memory-exceeds-reservation","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-47313927-36ed-4618-abae-9e2a400e4ff4.yaml","rules":[{"state":"firing","name":"SystemMemoryExceedsReservation","query":"sum by (node) (container_memory_rss{id=\"/system.slice\"}) \u003e ((sum by (node) (kube_node_status_capacity{resource=\"memory\"} - kube_node_status_allocatable{resource=\"memory\"})) * 0.95)","duration":900,"labels":{"namespace":"openshift-machine-config-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).","summary":"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved"},"alerts":[{"labels":{"alertname":"SystemMemoryExceedsReservation","namespace":"openshift-machine-config-operator","node":"fa-c2-39-02-35-20","severity":"warning"},"annotations":{"description":"System memory usage of 1.18G on fa-c2-39-02-35-20 exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).","summary":"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved"},"state":"firing","activeAt":"2024-03-26T21:32:38.8916566Z","value":"1.179844608e+09","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.00117222,"lastEvaluation":"2024-03-26T23:20:08.901383777Z","type":"alerting"}],"interval":30,"evaluationTime":0.001180486,"lastEvaluation":"2024-03-26T23:20:08.901379293Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"operator.marketplace.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-marketplace-marketplace-alert-rules-bc1b3701-c646-4cca-b840-4027dd66696f.yaml","rules":[{"state":"inactive","name":"OperatorHubSourceError","query":"catalogsource_ready{exported_namespace=\"openshift-marketplace\"} == 0","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Operators shipped via the {{ $labels.name }} source are not available for installation until the issue is fixed. Operators already installed from this source will not receive updates until issue is fixed. Inspect the status of the pod owned by {{ $labels.name }} source in the openshift-marketplace namespace (oc -n openshift-marketplace get pods -l olm.catalogSource={{ $labels.name }}) to diagnose and repair.","summary":"The {{ $labels.name }} source is in non-ready state for more than 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000232192,"lastEvaluation":"2024-03-26T23:19:48.937113746Z","type":"alerting"}],"interval":30,"evaluationTime":0.000240996,"lastEvaluation":"2024-03-26T23:19:48.93710773Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"alertmanager.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-alertmanager-main-rules-83f7de78-ab8e-4679-9f89-1f5befba7380.yaml","rules":[{"state":"inactive","name":"AlertmanagerClusterDown","query":"(count by (namespace, service) (avg_over_time(up{job=~\"alertmanager-main|alertmanager-user-workload\"}[5m]) \u003c 0.5) / count by (namespace, service) (up{job=~\"alertmanager-main|alertmanager-user-workload\"})) \u003e= 0.5","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value | humanizePercentage }} of Alertmanager instances within the {{$labels.job}} cluster have been up for less than half of the last 5m.","summary":"Half or more of the Alertmanager instances within the same cluster are down."},"alerts":[],"health":"ok","evaluationTime":0.000733094,"lastEvaluation":"2024-03-26T23:20:07.700281753Z","type":"alerting"},{"state":"inactive","name":"AlertmanagerClusterFailedToSendAlerts","query":"min by (namespace, service, integration) (rate(alertmanager_notifications_failed_total{integration=~\".*\",job=~\"alertmanager-main|alertmanager-user-workload\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{integration=~\".*\",job=~\"alertmanager-main|alertmanager-user-workload\"}[5m])) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The minimum notification failure rate to {{ $labels.integration }} sent from any instance in the {{$labels.job}} cluster is {{ $value | humanizePercentage }}.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerClusterFailedToSendAlerts.md","summary":"All Alertmanager instances in a cluster failed to send notifications to a critical integration."},"alerts":[],"health":"ok","evaluationTime":0.001953544,"lastEvaluation":"2024-03-26T23:20:07.697853239Z","type":"alerting"},{"state":"inactive","name":"AlertmanagerConfigInconsistent","query":"count by (namespace, service) (count_values by (namespace, service) (\"config_hash\", alertmanager_config_hash{job=~\"alertmanager-main|alertmanager-user-workload\"})) != 1","duration":1200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Alertmanager instances within the {{$labels.job}} cluster have different configurations.","summary":"Alertmanager instances within the same cluster have different configurations."},"alerts":[],"health":"ok","evaluationTime":0.000466739,"lastEvaluation":"2024-03-26T23:20:07.699810894Z","type":"alerting"},{"state":"inactive{"status":"success","data":{"groups":[{"name":"CloudCredentialOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-3026af6c-9fc0-4c0a-8742-ee4390976b54.yaml","rules":[{"state":"inactive","name":"CloudCredentialOperatorDeprovisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsDeprovisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be cleaned up","summary":"One or more CredentialsRequest CRs are unable to be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000139903,"lastEvaluation":"2024-03-26T23:20:12.635267096Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorInsufficientCloudCreds","query":"cco_credentials_requests_conditions{condition=\"InsufficientCloudCreds\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.","message":"Cluster's cloud credentials insufficient for minting or passthrough","summary":"Problem with the available platform credentials."},"alerts":[],"health":"ok","evaluationTime":0.000164241,"lastEvaluation":"2024-03-26T23:20:12.63540953Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorProvisioningFailed","query":"cco_credentials_requests_conditions{condition=\"CredentialsProvisionFailure\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.","message":"CredentialsRequest(s) unable to be fulfilled","summary":"One or more CredentialsRequest CRs are unable to be processed."},"alerts":[],"health":"ok","evaluationTime":0.00019769,"lastEvaluation":"2024-03-26T23:20:12.635061792Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorStaleCredentials","query":"cco_credentials_requests_conditions{condition=\"StaleCredentials\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.","message":"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).","summary":"One or more CredentialsRequest CRs are stale and should be deleted."},"alerts":[],"health":"ok","evaluationTime":0.000158249,"lastEvaluation":"2024-03-26T23:20:12.635576586Z","type":"alerting"},{"state":"inactive","name":"CloudCredentialOperatorTargetNamespaceMissing","query":"cco_credentials_requests_conditions{condition=\"MissingTargetNamespace\"} \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.","message":"CredentialsRequest(s) pointing to non-existent namespace","summary":"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace."},"alerts":[],"health":"ok","evaluationTime":0.000496007,"lastEvaluation":"2024-03-26T23:20:12.634561549Z","type":"alerting"}],"interval":30,"evaluationTime":0.001189226,"lastEvaluation":"2024-03-26T23:20:12.63455098Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-machine-approver.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-c62466a1-c2b8-49b3-bb8b-228428fdca6e.yaml","rules":[{"state":"inactive","name":"MachineApproverMaxPendingCSRsReached","query":"mapi_current_pending_csr \u003e mapi_max_pending_csr","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The number of pending CertificateSigningRequests has exceeded the\nmaximum threshold (current number of machine + 100). Check the\npending CSRs to determine which machines need approval, also check\nthat the nodelink controller is running in the openshift-machine-api\nnamespace.\n","summary":"max pending CSRs threshold reached."},"alerts":[],"health":"ok","evaluationTime":0.000600583,"lastEvaluation":"2024-03-26T23:20:07.013306911Z","type":"alerting"}],"interval":30,"evaluationTime":0.000627371,"lastEvaluation":"2024-03-26T23:20:07.013287571Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"node-tuning-operator.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-ac74f6d3-1240-4e6a-bf95-b3b636ad72f4.yaml","rules":[{"state":"inactive","name":"NTODegraded","query":"nto_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The Node Tuning Operator is degraded. Review the \"node-tuning\" ClusterOperator object for further details.","summary":"The Node Tuning Operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000261542,"lastEvaluation":"2024-03-26T23:19:55.261702925Z","type":"alerting"},{"state":"inactive","name":"NTOPodsNotReady","query":"kube_pod_status_ready{condition=\"true\",namespace=\"openshift-cluster-node-tuning-operator\"} == 0","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Pod {{ $labels.pod }} is not ready.\nReview the \"Event\" objects in \"openshift-cluster-node-tuning-operator\" namespace for further details.\n","summary":"Pod {{ $labels.pod }} is not ready."},"alerts":[],"health":"ok","evaluationTime":0.000550517,"lastEvaluation":"2024-03-26T23:19:55.261143441Z","type":"alerting"},{"name":"nto_custom_profiles:count","query":"count by (_id) (nto_profile_calculated_total{profile!~\"openshift\",profile!~\"openshift-control-plane\",profile!~\"openshift-node\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000257952,"lastEvaluation":"2024-03-26T23:19:55.261969025Z","type":"recording"}],"interval":30,"evaluationTime":0.001104755,"lastEvaluation":"2024-03-26T23:19:55.261126971Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"SamplesOperator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-7ec33583-2e3e-43fd-a61d-ac3004eb4e51.yaml","rules":[{"state":"inactive","name":"SamplesDegraded","query":"openshift_samples_degraded_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples could not be deployed and the operator is degraded. Review the \"openshift-samples\" ClusterOperator object for further details.\n","summary":"Samples operator is degraded."},"alerts":[],"health":"ok","evaluationTime":0.000101108,"lastEvaluation":"2024-03-26T23:20:14.798461722Z","type":"alerting"},{"state":"inactive","name":"SamplesImagestreamImportFailing","query":"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \u003e sum(openshift_samples_failed_imagestream_import_info)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports.  You can look at the \"openshift-samples\"\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is detecting problems with imagestream image imports"},"alerts":[],"health":"ok","evaluationTime":0.000380602,"lastEvaluation":"2024-03-26T23:20:14.798079339Z","type":"alerting"},{"state":"inactive","name":"SamplesInvalidConfig","query":"openshift_samples_invalidconfig_info == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator has been given an invalid configuration.\n","summary":"Samples operator Invalid configuration"},"alerts":[],"health":"ok","evaluationTime":0.000098191,"lastEvaluation":"2024-03-26T23:20:14.798564307Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingSecret","query":"openshift_samples_invalidsecret_info{reason=\"missing_secret\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator cannot find the samples pull secret in the openshift namespace.\n","summary":"Samples operator is not able to find secret"},"alerts":[],"health":"ok","evaluationTime":0.000102914,"lastEvaluation":"2024-03-26T23:20:14.798664107Z","type":"alerting"},{"state":"inactive","name":"SamplesMissingTBRCredential","query":"openshift_samples_invalidsecret_info{reason=\"missing_tbr_credential\"} == 1","duration":7200,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\n","summary":"Samples operator is not able to find the credentials for registry"},"alerts":[],"health":"ok","evaluationTime":0.00009672,"lastEvaluation":"2024-03-26T23:20:14.798768851Z","type":"alerting"},{"state":"inactive","name":"SamplesRetriesMissingOnImagestreamImportFailing","query":"sum(openshift_samples_failed_imagestream_import_info) \u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)","duration":7200,"labels":{"namespace":"openshift-cluster-samples-operator","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\nimports are not occurring.  Contact support.  You can look at the \"openshift-samples\" ClusterOperator object\nfor details. Most likely there are issues with the external image registry hosting the images that need to\nbe investigated.  The list of ImageStreams that have failing imports are:\n{{ range query \"openshift_samples_failed_imagestream_import_info \u003e 0\" }}\n  {{ .Labels.name }}\n{{ end }}\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\nretrying imports:\n{{ range query \"openshift_samples_retry_imagestream_import_total \u003e 0\" }}\n   {{ .Labels.imagestreamname }}\n{{ end }}\n","summary":"Samples operator is having problems with imagestream imports and its retries."},"alerts":[],"health":"ok","evaluationTime":0.000429624,"lastEvaluation":"2024-03-26T23:20:14.797648105Z","type":"alerting"},{"state":"inactive","name":"SamplesTBRInaccessibleOnBoot","query":"openshift_samples_tbr_inaccessible_info == 1","duration":172800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"One of two situations has occurred.  Either\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\nassociated with each image stream tag from the samples catalog is\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\nassist the mirroring process.\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\n","summary":"Samples operator is not able to access the registry on boot"},"alerts":[],"health":"ok","evaluationTime":0.000090898,"lastEvaluation":"2024-03-26T23:20:14.798866911Z","type":"alerting"}],"interval":30,"evaluationTime":0.001316085,"lastEvaluation":"2024-03-26T23:20:14.797643917Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"default-storage-classes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"MultipleDefaultStorageClasses","query":"max_over_time(default_storage_class_count[5m]) \u003e 1","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Cluster storage operator monitors all storage classes configured in the cluster\nand checks there is not more than one default StorageClass configured.\n","message":"StorageClass count check is failing (there should not be more than one default StorageClass)","summary":"More than one default StorageClass detected."},"alerts":[],"health":"ok","evaluationTime":0.000606995,"lastEvaluation":"2024-03-26T23:20:07.790381379Z","type":"alerting"}],"interval":30,"evaluationTime":0.000630893,"lastEvaluation":"2024-03-26T23:20:07.790363598Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-operations.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"state":"inactive","name":"PodStartupStorageOperationsFailing","query":"increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status!=\"success\"}[5m]) \u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\"volume_attach|volume_mount\",status=\"success\"}[5m]) == 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"Failing storage operation \"{{ $labels.operation_name }}\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\nfrom starting for past 5 minutes.\nPlease investigate Pods that are \"ContainerCreating\" on the node: \"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\".\nEvents of the Pods should contain exact error message: \"oc describe pod -n \u003cpod namespace\u003e \u003cpod name\u003e\".\n","summary":"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}."},"alerts":[],"health":"ok","evaluationTime":0.003327638,"lastEvaluation":"2024-03-26T23:19:49.969992393Z","type":"alerting"}],"interval":30,"evaluationTime":0.003350679,"lastEvaluation":"2024-03-26T23:19:49.969973817Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"storage-selinux.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-543f4e03-5b6c-4ab0-a66f-6df656414451.yaml","rules":[{"name":"cluster:volume_manager_selinux_pod_context_mismatch_total","query":"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000371888,"lastEvaluation":"2024-03-26T23:20:15.726454875Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_errors_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00012929,"lastEvaluation":"2024-03-26T23:20:15.726985553Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total","query":"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000154075,"lastEvaluation":"2024-03-26T23:20:15.726829056Z","type":"recording"},{"name":"cluster:volume_manager_selinux_volumes_admitted_total","query":"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\".*-e2e-.*\"})","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000088998,"lastEvaluation":"2024-03-26T23:20:15.727116332Z","type":"recording"}],"interval":30,"evaluationTime":0.000765861,"lastEvaluation":"2024-03-26T23:20:15.726443502Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-operators","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotEvaluateConditionalUpdates","query":"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\"Recommended\",status=\"Unknown\"}) \u003e= 3600)","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.","summary":"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000395125,"lastEvaluation":"2024-03-26T23:20:10.034645897Z","type":"alerting"},{"state":"inactive","name":"ClusterNotUpgradeable","query":"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\"Upgradeable\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"Upgradeable\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0) (ne (len (label \"reason\" $value)) 0) }}{{label \"reason\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000572266,"lastEvaluation":"2024-03-26T23:20:10.030396967Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDegraded","query":"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\"Degraded\",job=\"cluster-version-operator\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\"cluster-version-operator\"})) == 1)","duration":1800,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has been degraded for 30 minutes."},"alerts":[],"health":"ok","evaluationTime":0.00162457,"lastEvaluation":"2024-03-26T23:20:10.031935043Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorDown","query":"max by (namespace, name, reason) (cluster_operator_up{job=\"cluster-version-operator\"} == 0)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator has not been available for 10 minutes."},"alerts":[],"health":"ok","evaluationTime":0.000956651,"lastEvaluation":"2024-03-26T23:20:10.030974917Z","type":"alerting"},{"state":"inactive","name":"ClusterOperatorFlapping","query":"max by (namespace, name) (changes(cluster_operator_up{job=\"cluster-version-operator\"}[2m]) \u003e 2)","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster operator up status is changing often."},"alerts":[],"health":"ok","evaluationTime":0.001077533,"lastEvaluation":"2024-03-26T23:20:10.033563948Z","type":"alerting"}],"interval":30,"evaluationTime":0.004665296,"lastEvaluation":"2024-03-26T23:20:10.030380586Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"cluster-version","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-5537b364-2011-4734-8a76-40acff4f25bf.yaml","rules":[{"state":"inactive","name":"CannotRetrieveUpdates","query":"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\"RetrievedUpdates\",endpoint=\"metrics\",name=\"version\",reason!=\"NoChannel\"}))","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \"cluster_operator_conditions\" | query}}{{range $value := .}}{{if and (eq (label \"name\" $value) \"version\") (eq (label \"condition\" $value) \"RetrievedUpdates\") (eq (label \"endpoint\" $value) \"metrics\") (eq (value $value) 0.0)}}{{label \"reason\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\n'}\"`{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}."},"alerts":[],"health":"ok","evaluationTime":0.000776039,"lastEvaluation":"2024-03-26T23:20:07.559957604Z","type":"alerting"},{"state":"inactive","name":"ClusterReleaseNotAccepted","query":"max by (namespace, name, reason) (cluster_operator_conditions{condition=\"ReleaseAccepted\",endpoint=\"metrics\",name=\"version\"} == 0)","duration":3600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"The desired cluster release has not been accepted for at least an hour."},"alerts":[],"health":"ok","evaluationTime":0.000278286,"lastEvaluation":"2024-03-26T23:20:07.561683145Z","type":"alerting"},{"state":"inactive","name":"ClusterVersionOperatorDown","query":"absent(up{job=\"cluster-version-operator\"} == 1)","duration":600,"labels":{"namespace":"openshift-cluster-version","prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} For more information refer to {{ label \"url\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}","summary":"Cluster version operator has disappeared from Prometheus target discovery."},"alerts":[],"health":"ok","evaluationTime":0.000543782,"lastEvaluation":"2024-03-26T23:20:07.559408749Z","type":"alerting"},{"state":"firing","name":"UpdateAvailable","query":"sum by (channel, namespace, upstream) (cluster_version_available_updates) \u003e 0","duration":0,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"info"},"annotations":{"description":"For more information refer to 'oc adm upgrade'{{ with $console_url := \"console_url\" | query }}{{ if ne (len (label \"url\" (first $console_url ) ) ) 0}} or {{ label \"url\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.","summary":"Your upstream update recommendation service recommends you update your cluster."},"alerts":[{"labels":{"alertname":"UpdateAvailable","channel":"stable-4.15","namespace":"openshift-cluster-version","severity":"info","upstream":"\u003cdefault\u003e"},"annotations":{"description":"For more information refer to 'oc adm upgrade' or https://console-openshift-console.apps.cicd-190b9d7ec1.ocp-dev-ppc64le.com/settings/cluster/.","summary":"Your upstream update recommendation service recommends you update your cluster."},"state":"firing","activeAt":"2024-03-26T21:11:37.55816001Z","value":"1e+00","partialResponseStrategy":"WARN"}],"health":"ok","evaluationTime":0.000941147,"lastEvaluation":"2024-03-26T23:20:07.560738393Z","type":"alerting"}],"interval":30,"evaluationTime":0.002574482,"lastEvaluation":"2024-03-26T23:20:07.55939199Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift/console-operator","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-c9d370fc-e2d2-4928-ba69-38329b1a5c11.yaml","rules":[{"name":"cluster:console_auth_login_failures_total:sum","query":"sum by (reason) (console_auth_login_failures_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000168884,"lastEvaluation":"2024-03-26T23:20:06.448469806Z","type":"recording"},{"name":"cluster:console_auth_login_requests_total:sum","query":"sum(console_auth_login_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.00034932,"lastEvaluation":"2024-03-26T23:20:06.447961034Z","type":"recording"},{"name":"cluster:console_auth_login_successes_total:sum","query":"sum by (role) (console_auth_login_successes_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000150669,"lastEvaluation":"2024-03-26T23:20:06.448315977Z","type":"recording"},{"name":"cluster:console_auth_logout_requests_total:sum","query":"sum by (reason) (console_auth_logout_requests_total)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000155632,"lastEvaluation":"2024-03-26T23:20:06.448642331Z","type":"recording"},{"name":"cluster:console_customization_perspectives_info:max","query":"max by (name, state) (console_customization_perspectives_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000130589,"lastEvaluation":"2024-03-26T23:20:06.44927447Z","type":"recording"},{"name":"cluster:console_plugins_info:max","query":"max by (name, state) (console_plugins_info)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000303309,"lastEvaluation":"2024-03-26T23:20:06.448966573Z","type":"recording"},{"name":"cluster:console_usage_users:max","query":"max by (role) (console_usage_users)","labels":{"prometheus":"openshift-monitoring/k8s"},"health":"ok","evaluationTime":0.000159865,"lastEvaluation":"2024-03-26T23:20:06.448802324Z","type":"recording"}],"interval":30,"evaluationTime":0.001461473,"lastEvaluation":"2024-03-26T23:20:06.447947766Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"openshift-dns.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-19d1a779-16b0-4a4b-8300-e46827430561.yaml","rules":[{"state":"inactive","name":"CoreDNSErrorsHigh","query":"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\"SERVFAIL\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \u003e 0.01","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.","summary":"CoreDNS serverfail"},"alerts":[],"health":"ok","evaluationTime":0.000587757,"lastEvaluation":"2024-03-26T23:20:01.250136252Z","type":"alerting"},{"state":"inactive","name":"CoreDNSHealthCheckSlow","query":"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \u003e 10","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})","summary":"CoreDNS health checks"},"alerts":[],"health":"ok","evaluationTime":0.000601462,"lastEvaluation":"2024-03-26T23:20:01.249530348Z","type":"alerting"},{"state":"inactive","name":"CoreDNSPanicking","query":"increase(coredns_panics_total[10m]) \u003e 0","duration":300,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}","summary":"CoreDNS panic"},"alerts":[],"health":"ok","evaluationTime":0.000550415,"lastEvaluation":"2024-03-26T23:20:01.24897421Z","type":"alerting"}],"interval":30,"evaluationTime":0.001773205,"lastEvaluation":"2024-03-26T23:20:01.248955999Z","limit":0,"partialResponseStrategy":"ABORT"},{"name":"etcd","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-4c592245-7443-40fa-b864-b42f2ddb7a33.yaml","rules":[{"state":"inactive","name":"etcdDatabaseHighFragmentationRatio","query":"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \u003e 1.048576e+08","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"alerts":[],"health":"ok","evaluationTime":0.000499448,"lastEvaluation":"2024-03-26T23:20:08.901384855Z","type":"alerting"},{"state":"inactive","name":"etcdDatabaseQuotaLowSpace","query":"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \u003e 95","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"critical"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.","runbook_url":"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md","summary":"etcd cluster database is running full."},"alerts":[],"health":"ok","evaluationTime":0.000431617,"lastEvaluation":"2024-03-26T23:20:08.900463094Z","type":"alerting"},{"state":"inactive","name":"etcdExcessiveDatabaseGrowth","query":"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \u003e etcd_server_quota_backend_bytes","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.","summary":"etcd cluster database growing very fast."},"alerts":[],"health":"ok","evaluationTime":0.000483318,"lastEvaluation":"2024-03-26T23:20:08.900898475Z","type":"alerting"},{"state":"inactive","name":"etcdHighCommitDurations","query":"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\".*etcd.*\"}[5m])) \u003e 0.25","duration":600,"labels":{"prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"etcd cluster \"{{ $labels.job }}\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.","summary":"etcd cluster 99th percentile commit durations are too high."},"alerts":[],"health":"ok","evaluationTime":0.000695465,"lastEvaluation":"202
Ginkgo exit error 1: exit with code 1
pull-ci-openshift-installer-master-e2e-aws-ovn-fips (all) - 8 runs, 0% failed, 13% of runs match
#1772724235886661632junit39 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10m6s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10m6s, firing for 0s:
Mar 26 21:45:57.443 - 214s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 21:45:57.443 - 244s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 21:51:33.443 - 148s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-metal-ipi-sdn-serial-ipv4 (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
#1772707772761116672junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m6s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m6s, firing for 0s:
Mar 26 21:00:02.094 - 186s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.14-e2e-aws-sdn-techpreview (all) - 5 runs, 0% failed, 20% of runs match
#1772730519503507456junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m52s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m52s, firing for 0s:
Mar 26 21:36:35.030 - 112s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-altinfra-e2e-aws-ovn-wavelengthzones (all) - 9 runs, 67% failed, 33% of failures match = 22% impact
#1772724226030047232junit40 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772724226030047232junit40 hours ago
        <*errors.errorString | 0xc002569150>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.15:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6cb844dc65-mpdxk\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711489117.212,\n      \"1\"\n    ]\n  }\n]",
        },
#1772724226030047232junit40 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772724226030047232junit40 hours ago
        <*errors.errorString | 0xc0021573a0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithNoRunningPhase\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-i0yhxlvp-e9ff3-st7bq-edge-us-west-2-wl1-las-wlz-1-jgrkv\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711492353.13,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithoutValidNode\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-i0yhxlvp-e9ff3-st7bq-edge-us-west-2-wl1-las-wlz-1-jgrkv\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711492353.13,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.129.0.15:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-6cb844dc65-mpdxk\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711492353.13,\n      \"1\"\n    ]\n  }\n]",
        },
#1772667058517970944junit43 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772667058517970944junit43 hours ago
        <*errors.errorString | 0xc001caf100>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.16:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-574b9ccd7-p44mf\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711475823.727,\n      \"1\"\n    ]\n  }\n]",
        },
#1772667058517970944junit43 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772667058517970944junit43 hours ago
        <*errors.errorString | 0xc0019b51d0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithNoRunningPhase\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-wit2920d-e9ff3-bv9r8-edge-us-east-1-wl1-dfw-wlz-1-wlc8w\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711478822.238,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"MachineWithoutValidNode\",\n      \"alertstate\": \"firing\",\n      \"name\": \"ci-op-wit2920d-e9ff3-bv9r8-edge-us-east-1-wl1-dfw-wlz-1-wlc8w\",\n      \"namespace\": \"openshift-machine-api\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711478822.238,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"TechPreviewNoUpgrade\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-apiserver-operator\",\n      \"endpoint\": \"https\",\n      \"instance\": \"10.128.0.16:8443\",\n      \"job\": \"metrics\",\n      \"name\": \"CustomNoUpgrade\",\n      \"namespace\": \"openshift-kube-apiserver-operator\",\n      \"pod\": \"kube-apiserver-operator-574b9ccd7-p44mf\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711478822.238,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-release-master-nightly-4.12-e2e-ibmcloud-csi (all) - 1 runs, 0% failed, 100% of runs match
#1772736658207150080junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m10s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 12m10s, firing for 0s:
Mar 26 22:16:11.330 - 124s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 22:16:11.330 - 606s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.14-e2e-azure-csi (all) - 5 runs, 0% failed, 20% of runs match
#1772730525392310272junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 26 22:04:07.326 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-openstack-ovn (all) - 6 runs, 33% failed, 50% of failures match = 17% impact
#1772714095254966272junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m26s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"openstack", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m26s, firing for 0s:
Mar 26 21:05:41.955 - 266s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-kubernetes-autoscaler-release-4.12-e2e-aws (all) - 5 runs, 0% failed, 20% of runs match
#1772723145371815936junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m40s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 1m40s, firing for 0s:
Mar 26 21:20:16.972 - 100s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-release-4.13-e2e-aws-sdn-network-migration-rollback (all) - 2 runs, 100% failed, 50% of failures match = 50% impact
#1772708899485388800junit40 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m30s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 6m30s, firing for 0s:
Mar 26 21:19:13.218 - 390s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
rehearse-44057-periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-gcp-ovn-rt-upgrade (all) - 4 runs, 75% failed, 33% of failures match = 25% impact
#1772691036443774976junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.15", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m58s, firing for 0s:
Mar 26 19:39:02.635 - 478s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-azure-sdn-techpreview-serial (all) - 4 runs, 75% failed, 33% of failures match = 25% impact
#1772707726695075840junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 24m14s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 24m14s, firing for 0s:
Mar 26 20:18:49.874 - 218s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 20:18:49.874 - 1118s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 20:38:59.874 - 118s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-hybrid-step-registry (all) - 6 runs, 17% failed, 100% of failures match = 17% impact
#1772714096932687872junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m46s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m46s, firing for 0s:
Mar 26 20:38:39.756 - 286s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
release-openshift-ocp-installer-e2e-azure-serial-4.13 (all) - 4 runs, 0% failed, 25% of runs match
#1772707725533253632junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m8s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m8s, firing for 0s:
Mar 26 20:15:03.166 - 68s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-ovn (all) - 6 runs, 33% failed, 50% of failures match = 17% impact
#1772714101554810880junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 26 20:47:29.881 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-nightly-4.13-ocp-e2e-aws-ovn-heterogeneous-upgrade (all) - 1 runs, 0% failed, 100% of runs match
#1772703188177129472junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h45m40s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h45m40s, firing for 0s:
Mar 26 20:04:17.616 - 1232s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 20:04:17.616 - 5040s I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 20:26:21.616 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 21:29:49.616 - 10s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-network-operator-release-4.13-e2e-aws-sdn-network-reverse-migration (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1772708899502166016junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 28s, firing for 0s:
Mar 26 20:47:02.206 - 28s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-ovn-multi (all) - 3 runs, 0% failed, 33% of runs match
#1772704445453307904junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 1m28s, firing for 0s:
Mar 26 20:41:53.152 - 88s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-okd-scos-4.14-e2e-aws-ovn (all) - 1 runs, 0% failed, 100% of runs match
#1772714765089509376junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m54s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m54s, firing for 0s:
Mar 26 20:36:33.761 - 294s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
rehearse-50250-periodic-ci-openshift-ovn-kubernetes-master-e2e-ibmcloud-ipi-ovn-periodic (all) - 2 runs, 0% failed, 100% of runs match
#1772698080496324608junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 28m54s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 28m54s, firing for 0s:
Mar 26 20:10:39.649 - 988s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 20:11:03.649 - 328s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 26 20:11:09.649 - 418s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
#1772657136489205760junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 44m54s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 44m54s, firing for 0s:
Mar 26 17:28:15.451 - 388s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 17:28:15.451 - 2098s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 17:28:35.451 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-aws-sdn-upgrade (all) - 6 runs, 17% failed, 100% of failures match = 17% impact
#1772691724976525312junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m2s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 4m2s, firing for 0s:
Mar 26 19:05:40.609 - 242s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn (all) - 3 runs, 0% failed, 33% of runs match
#1772708284231323648junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m4s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m4s, firing for 0s:
Mar 26 20:16:42.973 - 2s    I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 20:16:42.973 - 62s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-stable-4.12-ocp-e2e-aws-sdn-arm64 (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
#1772674209261031424junit41 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m32s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 3m32s, firing for 0s:
Mar 26 17:51:53.126 - 212s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-e2e-network-migration-rollback (all) - 2 runs, 0% failed, 50% of runs match
#1772690352528953344junit42 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m26s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=2s): pending for 4m26s, firing for 0s:
Mar 26 19:58:30.095 - 266s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-stable-4.14-ocp-e2e-aws-sdn-arm64 (all) - 8 runs, 63% failed, 20% of failures match = 13% impact
#1772685658440077312junit42 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m26s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m26s, firing for 0s:
Mar 26 18:40:08.197 - 146s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-sdn-upgrade (all) - 2 runs, 0% failed, 50% of runs match
#1772668973465210880junit42 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m44s on platformidentification.JobType{Release:"4.12", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 5m44s, firing for 0s:
Mar 26 18:46:22.534 - 344s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.12-upgrade-from-stable-4.11-e2e-metal-ipi-upgrade-ovn-ipv6 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772668913838985216junit42 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 51m50s on platformidentification.JobType{Release:"4.12", FromRelease:"4.11", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 51m50s, firing for 0s:
Mar 26 20:00:35.108 - 1436s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 20:01:35.108 - 1376s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 20:01:47.108 - 298s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-release-master-nightly-4.12-e2e-metal-ipi-ovn-dualstack-local-gateway (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1772683557743890432junit42 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 20m52s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 20m52s, firing for 0s:
Mar 26 20:03:19.777 - 1252s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-ovn-kubernetes-master-4.16-upgrade-from-stable-4.15-e2e-gcp-ovn-rt-upgrade (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
#1772638381541429248junit42 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772638381541429248junit42 hours ago
        <*errors.errorString | 0xc002951640>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeDeploymentReplicasMismatch\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-main\",\n      \"deployment\": \"ovnkube-control-plane\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711478678.587,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"ovnkube-cluster-manager\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"pod\": \"ovnkube-control-plane-844ccd8879-f8ccg\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"dcf6b0e5-2b51-485c-bbd6-a083cb16de03\"\n    },\n    \"value\": [\n      1711478678.587,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"ovnkube-cluster-manager\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"pod\": \"ovnkube-control-plane-844ccd8879-lm52n\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"162cb116-cde5-405a-a3a3-9b4d08c6d3cb\"\n    },\n    \"value\": [\n      1711478678.587,\n      \"1\"\n    ]\n  }\n]",
        },
#1772638381541429248junit42 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772638381541429248junit42 hours ago
        <*errors.errorString | 0xc001e49fc0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"ClusterOperatorDegraded\",\n      \"alertstate\": \"firing\",\n      \"name\": \"network\",\n      \"namespace\": \"openshift-cluster-version\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"RolloutHung\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711481828.345,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeDeploymentReplicasMismatch\",\n      \"alertstate\": \"firing\",\n      \"container\": \"kube-rbac-proxy-main\",\n      \"deployment\": \"ovnkube-control-plane\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711481828.345,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"ovnkube-cluster-manager\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"pod\": \"ovnkube-control-plane-844ccd8879-f8ccg\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"dcf6b0e5-2b51-485c-bbd6-a083cb16de03\"\n    },\n    \"value\": [\n      1711481828.345,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"ovnkube-cluster-manager\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"pod\": \"ovnkube-control-plane-844ccd8879-lm52n\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"162cb116-cde5-405a-a3a3-9b4d08c6d3cb\"\n    },\n    \"value\": [\n      1711481828.345,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-ovn-serial-aws-arm64 (all) - 4 runs, 25% failed, 200% of failures match = 50% impact
#1772674207658807296junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6s, firing for 0s:
Mar 26 18:00:03.880 - 6s    I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772632250773409792junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m48s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m48s, firing for 0s:
Mar 26 15:13:16.628 - 108s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-azure-upgrade (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
#1772665674649309184junit43 hours ago
[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
[bz-storage][invariant] alert/KubePersistentVolumeErrors should not be at or above pending
#1772665674649309184junit43 hours ago
[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
[bz-storage][invariant] alert/KubePersistentVolumeErrors should not be at or above pending
#1772665674649309184junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 14m55s on platformidentification.JobType{Release:"4.10", FromRelease:"4.9", Platform:"azure", Network:"sdn", Topology:"ha"} (maxAllowed=3s): pending for 14m55s, firing for 0s:
Mar 26 18:40:53.137 - 895s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772665674649309184junit43 hours ago
# [sig-arch][bz-kube-apiserver][Late] Alerts alert/KubeAPIErrorBudgetBurn should not be at or above pending [Suite:openshift/conformance/parallel]
flake: KubeAPIErrorBudgetBurn was at or above pending for at least 15m55s on platformidentification.JobType{Release:"4.10", FromRelease:"4.9", Platform:"azure", Network:"sdn", Topology:"ha"} (maxAllowed=3s): pending for 15m55s, firing for 0s:
#1772618564285501440junit46 hours ago
[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
[bz-storage][invariant] alert/KubePersistentVolumeErrors should not be at or above pending
#1772618564285501440junit46 hours ago
[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
[bz-storage][invariant] alert/KubePersistentVolumeErrors should not be at or above pending
#1772618564285501440junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m26s on platformidentification.JobType{Release:"4.10", FromRelease:"4.9", Platform:"azure", Network:"sdn", Topology:"ha"} (maxAllowed=3s): pending for 12m26s, firing for 0s:
Mar 26 14:32:42.274 - 1s    I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 15:43:25.274 - 239s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 15:43:25.274 - 506s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.16-e2e-openstack-ovn-password (all) - 2 runs, 0% failed, 50% of runs match
#1772673743210942464junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 4m32s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"openstack", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 4m32s, firing for 0s:
Mar 26 18:18:32.131 - 272s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.12-ocp-e2e-aws-ovn-arm64-techpreview-serial (all) - 1 runs, 0% failed, 100% of runs match
#1772655271454183424junit43 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 44s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 44s, firing for 0s:
Mar 26 16:38:28.900 - 44s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-ovn-proxy (all) - 2 runs, 0% failed, 50% of runs match
#1772668914703011840junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m20s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 1m20s, firing for 0s:
Mar 26 17:38:40.625 - 80s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.12-upgrade-from-stable-4.11-ocp-e2e-aws-sdn-arm64 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772655271580012544junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m52s on platformidentification.JobType{Release:"4.12", FromRelease:"4.11", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 9m52s, firing for 0s:
Mar 26 16:39:45.155 - 592s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-vmware-vsphere-csi-driver-master-e2e-vsphere-ovn-upgrade (all) - 4 runs, 100% failed, 25% of failures match = 25% impact
#1772644348433797120junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 12m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 12m58s, firing for 0s:
Mar 26 16:13:31.255 - 778s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-ovn-upi (all) - 1 runs, 0% failed, 100% of runs match
#1772668975159709696junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m16s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 9m16s, firing for 0s:
Mar 26 17:47:37.278 - 98s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 17:47:37.278 - 458s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.12-e2e-vsphere-ovn-upi-serial (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772668928942673920junit44 hours ago
        <*errors.errorString | 0xc002149e30>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubeAggregatedAPIErrors\",\n      \"alertstate\": \"firing\",\n      \"name\": \"v1beta1.metrics.k8s.io\",\n      \"namespace\": \"default\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711474489.234,\n      \"1\"\n    ]\n  }\n]",
        },
#1772668928942673920junit44 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-heterogeneous (all) - 6 runs, 17% failed, 100% of failures match = 17% impact
#1772670411264233472junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m48s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m48s, firing for 0s:
Mar 26 17:45:16.079 - 108s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.12-ocp-e2e-ovn-serial-aws-arm64 (all) - 1 runs, 0% failed, 100% of runs match
#1772655271391268864junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m18s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 1m18s, firing for 0s:
Mar 26 16:44:03.076 - 78s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn (all) - 3 runs, 0% failed, 33% of runs match
#1772668931526365184junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m40s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=2s): pending for 8m40s, firing for 0s:
Mar 26 17:36:12.843 - 80s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 17:36:12.843 - 440s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.12-ocp-e2e-sdn-serial-aws-arm64 (all) - 1 runs, 0% failed, 100% of runs match
#1772655271663898624junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 42s on platformidentification.JobType{Release:"4.12", FromRelease:"", Platform:"aws", Architecture:"arm64", Network:"sdn", Topology:"ha"} (maxAllowed=2s): pending for 42s, firing for 0s:
Mar 26 16:34:40.530 - 42s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.15-upgrade-from-stable-4.14-e2e-aws-sdn-upgrade (all) - 5 runs, 20% failed, 100% of failures match = 20% impact
#1772647031974662144junit44 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 24s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 24s, firing for 0s:
Mar 26 16:05:04.340 - 24s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.11-e2e-gcp-upgrade (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
#1772636061533474816junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 51s on platformidentification.JobType{Release:"4.11", FromRelease:"4.11", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 51s, firing for 0s:
Mar 26 15:26:32.296 - 51s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-aws-ovn-upgrade (all) - 3 runs, 0% failed, 33% of runs match
#1772632233929084928junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m46s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m46s, firing for 0s:
Mar 26 15:06:50.604 - 466s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-serial-aws-ovn-heterogeneous (all) - 5 runs, 20% failed, 200% of failures match = 40% impact
#1772632661915865088junit45 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m28s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m28s, firing for 0s:
Mar 26 15:19:39.932 - 328s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772617319810338816junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h5m12s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h5m12s, firing for 0s:
Mar 26 14:23:17.588 - 352s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 14:23:17.588 - 3532s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 15:23:41.588 - 28s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-installer-master-e2e-vsphere-upi-zones (all) - 5 runs, 60% failed, 33% of failures match = 20% impact
#1772631662039928832junit46 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info",namespace!="openshift-e2e-loki"} >= 1
    [
#1772631662039928832junit46 hours ago
        <*errors.errorString | 0xc000b5cf80>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\",namespace!=\"openshift-e2e-loki\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"VSphereOpenshiftClusterHealthFail\",\n      \"alertstate\": \"firing\",\n      \"check\": \"CheckAccountPermissions\",\n      \"container\": \"vsphere-problem-detector-operator\",\n      \"endpoint\": \"vsphere-metrics\",\n      \"instance\": \"10.129.0.14:8444\",\n      \"job\": \"vsphere-problem-detector-metrics\",\n      \"namespace\": \"openshift-cluster-storage-operator\",\n      \"pod\": \"vsphere-problem-detector-operator-769569896-r7n9j\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"vsphere-problem-detector-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711466918.175,\n      \"1\"\n    ]\n  }\n]",
        },
pull-ci-openshift-ibm-vpc-block-csi-driver-master-e2e-ibmcloud-csi (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772644271334100992junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 24m18s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 24m18s, firing for 0s:
Mar 26 16:21:44.922 - 6s    I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 16:21:44.922 - 642s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 16:22:22.922 - 508s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 16:23:00.922 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
Mar 26 16:31:22.922 - 64s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
pull-ci-openshift-installer-master-e2e-vsphere-ovn (all) - 5 runs, 40% failed, 50% of failures match = 20% impact
#1772631661960237056junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 8m28s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 8m28s, firing for 0s:
Mar 26 15:38:24.007 - 508s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-gcp-ovn-upgrade (all) - 4 runs, 0% failed, 25% of runs match
#1772617082345623552junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m42s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m42s, firing for 0s:
Mar 26 14:07:36.284 - 314s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 14:14:22.284 - 268s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.15-ocp-e2e-upgrade-azure-ovn-arm64 (all) - 8 runs, 13% failed, 100% of failures match = 13% impact
#1772632778387492864junit46 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info"} >= 1
    [
#1772632778387492864junit46 hours ago
        <*errors.errorString | 0xc0020eb190>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"KubePodCrashLooping\",\n      \"alertstate\": \"firing\",\n      \"container\": \"console\",\n      \"endpoint\": \"https-main\",\n      \"job\": \"kube-state-metrics\",\n      \"namespace\": \"openshift-console\",\n      \"pod\": \"console-64d445cb8c-thtw6\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"reason\": \"CrashLoopBackOff\",\n      \"service\": \"kube-state-metrics\",\n      \"severity\": \"warning\",\n      \"uid\": \"ca40559a-6c97-4bf2-8f92-0cbdf2721d59\"\n    },\n    \"value\": [\n      1711468234.858,\n      \"1\"\n    ]\n  }\n]",
        },
periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-sdn-upgrade (all) - 5 runs, 0% failed, 20% of runs match
#1772619014019747840junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m4s on platformidentification.JobType{Release:"4.14", FromRelease:"4.13", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 5m4s, firing for 0s:
Mar 26 14:12:15.393 - 276s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 14:18:23.393 - 28s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-gcp-sdn-upgrade (all) - 4 runs, 0% failed, 25% of runs match
#1772617134686343168junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m26s on platformidentification.JobType{Release:"4.13", FromRelease:"4.12", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m26s, firing for 0s:
Mar 26 14:06:35.594 - 86s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-capi-operator-main-e2e-azure-ovn-techpreview (all) - 2 runs, 0% failed, 50% of runs match
#1772626029005770752junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 58s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 58s, firing for 0s:
Mar 26 15:13:19.544 - 58s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1772613843827036160junit46 hours ago
[bz-etcd][invariant] alert/etcdMemberCommunicationSlow should not be at or above pending
[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
[sig-arch] Monitor cluster while tests execute
#1772613843827036160junit46 hours ago
[bz-etcd][invariant] alert/etcdMemberCommunicationSlow should not be at or above pending
[bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
[sig-arch] Monitor cluster while tests execute
#1772613843827036160junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h9m15s on platformidentification.JobType{Release:"4.11", FromRelease:"4.10", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1h9m15s, firing for 0s:
Mar 26 14:11:45.568 - 4155s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772613843827036160junit46 hours ago
# [sig-arch][bz-kube-apiserver][Late] Alerts alert/KubeAPIErrorBudgetBurn should not be at or above pending [Suite:openshift/conformance/parallel]
flake: KubeAPIErrorBudgetBurn was at or above pending for at least 56s on platformidentification.JobType{Release:"4.11", FromRelease:"4.10", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 56s, firing for 0s:
periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-stable-4.14-ocp-e2e-upgrade-gcp-ovn-arm64 (all) - 8 runs, 13% failed, 100% of failures match = 13% impact
#1772632778320384000junit46 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 3m28s on platformidentification.JobType{Release:"4.15", FromRelease:"4.14", Platform:"gcp", Architecture:"arm64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 3m28s, firing for 0s:
Mar 26 15:09:31.350 - 208s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-cluster-capi-operator-main-e2e-aws-ovn-techpreview (all) - 1 runs, 0% failed, 100% of runs match
#1772626028968022016junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 5m22s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 5m22s, firing for 0s:
Mar 26 14:51:04.991 - 322s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-aws-ovn-serial (all) - 1 runs, 0% failed, 100% of runs match
#1772615357874311168junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m40s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1m40s, firing for 0s:
Mar 26 14:03:37.709 - 100s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-aws-upgrade-ovn-single-node (all) - 1 runs, 0% failed, 100% of runs match
#1772615361443663872junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m28s on platformidentification.JobType{Release:"4.13", FromRelease:"4.13", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"single"} (maxAllowed=0s): pending for 1m28s, firing for 0s:
Mar 26 14:05:41.993 - 88s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-release-master-ci-4.13-e2e-azure-sdn-serial (all) - 1 runs, 0% failed, 100% of runs match
#1772615364014772224junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m26s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m26s, firing for 0s:
Mar 26 14:12:29.759 - 86s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-ovn-serial (all) - 1 runs, 0% failed, 100% of runs match
#1772615376404746240junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 10s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 10s, firing for 0s:
Mar 26 14:04:07.893 - 10s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-aws-ovn-serial (all) - 21 runs, 19% failed, 50% of failures match = 10% impact
#1772605317633282048junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m42s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m42s, firing for 0s:
Mar 26 13:45:08.635 - 402s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
#1772596807793119232junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 47m12s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 47m12s, firing for 0s:
Mar 26 13:01:59.063 - 368s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 13:01:59.063 - 2018s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 13:09:39.063 - 28s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 13:37:09.063 - 418s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.11-e2e-aws-ovn-upgrade (all) - 1 runs, 0% failed, 100% of runs match
#1772613843743150080junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 6m25s on platformidentification.JobType{Release:"4.11", FromRelease:"4.11", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 6m25s, firing for 0s:
Mar 26 13:50:14.960 - 116s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 13:53:41.960 - 269s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.13-e2e-network-migration-rollback (all) - 1 runs, 0% failed, 100% of runs match
#1772615368016138240junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m4s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m4s, firing for 0s:
Mar 26 14:59:48.904 - 124s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-metal-ipi-ovn-dualstack (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1772617076469403648junit47 hours ago
        <*errors.errorString | 0xc001ced3f0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseClusterMemberError\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464943.105,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseInboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464943.105,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OVNKubernetesSouthboundDatabaseOutboundConnectionMissing\",\n      \"alertstate\": \"firing\",\n      \"namespace\": \"openshift-ovn-kubernetes\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1711464943.105,\n      \"1\"\n    ]\n  }\n]",
        },
#1772617076469403648junit47 hours ago
    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing",alertstate="firing",severity!="info"} >= 1
    [
periodic-ci-openshift-release-master-ci-4.14-e2e-gcp-ovn (all) - 5 runs, 0% failed, 20% of runs match
#1772619053446205440junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 16m28s on platformidentification.JobType{Release:"4.14", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 16m28s, firing for 0s:
Mar 26 14:17:57.129 - 16s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}
Mar 26 14:17:57.129 - 22s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 14:17:57.129 - 892s  I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 14:34:21.129 - 58s   I alert/KubeAPIErrorBudgetBurn namespace/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-ovn-proxy (all) - 4 runs, 0% failed, 25% of runs match
#1772617154890305536junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m2s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m2s, firing for 0s:
Mar 26 14:26:29.737 - 122s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-metal-ipi-sdn (all) - 16 runs, 69% failed, 9% of failures match = 6% impact
#1772598927078461440junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1h24m46s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"metal", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1h24m46s, firing for 0s:
Mar 26 14:06:42.348 - 3860s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 14:07:42.348 - 988s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 14:09:00.348 - 238s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/critical ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}
periodic-ci-openshift-release-master-ci-4.13-e2e-gcp-sdn-techpreview (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1772617112473309184junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m8s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"gcp", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 2m8s, firing for 0s:
Mar 26 14:13:45.220 - 128s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-ovn-remote-libvirt-ppc64le (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
#1772624692784402432junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 2m14s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"libvirt", Architecture:"ppc64le", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 2m14s, firing for 0s:
Mar 26 14:31:55.686 - 134s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-sdn-cgroupsv2 (all) - 4 runs, 0% failed, 25% of runs match
#1772617109935755264junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 52s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 52s, firing for 0s:
Mar 26 14:12:11.658 - 52s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-origin-master-e2e-aws-ovn-fips (all) - 22 runs, 32% failed, 14% of failures match = 5% impact
#1772605317574561792junit47 hours ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 9m16s on platformidentification.JobType{Release:"4.16", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m16s, firing for 0s:
Mar 26 13:50:11.229 - 556s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-console-operator-master-e2e-azure-ovn-upgrade (all) - 2 runs, 0% failed, 50% of runs match
#1772603391613079552junit2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 23m56s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 23m56s, firing for 0s:
Mar 26 13:39:09.183 - 508s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
Mar 26 14:03:39.183 - 928s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-sdn-crun (all) - 1 runs, 0% failed, 100% of runs match
#1772615609473830912junit2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 1m38s on platformidentification.JobType{Release:"4.13", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 1m38s, firing for 0s:
Mar 26 13:58:26.446 - 98s   I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade (all) - 4 runs, 50% failed, 50% of failures match = 25% impact
#1772593622709964800junit2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 7m28s on platformidentification.JobType{Release:"4.12", FromRelease:"4.12", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 7m28s, firing for 0s:
Mar 26 12:33:11.143 - 448s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-aws-sdn-upgrade (all) - 2 runs, 0% failed, 50% of runs match
#1772593622672216064junit2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 35m40s on platformidentification.JobType{Release:"4.12", FromRelease:"4.11", Platform:"aws", Architecture:"amd64", Network:"sdn", Topology:"ha"} (maxAllowed=0s): pending for 35m40s, firing for 0s:
Mar 26 12:30:30.992 - 140s  I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Mar 26 12:30:30.992 - 2000s I alert/KubeAPIErrorBudgetBurn ns/openshift-kube-apiserver ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}
pull-ci-openshift-machine-api-operator-master-e2e-vsphere-ovn-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1772604655621115904junit2 days ago
# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 19m58s on platformidentification.JobType{Release:"4.16", FromRelease:"4.16", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 19m58s, firing for 0s:
Mar 26 13:33:05.852 - 1198s I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}

Found in 1.32% of runs (7.90% of failures) across 45405 total runs and 5166 jobs (16.70% failed) in 3.326s - clear search | chart view - source code located on github