We have many guards making sure that there are always at least two instances of the kube-apiserver. If we ever reach a single kube-apiserver and it causes disruption for the clients, other alerts such as KubeAPIErrorBudgetBurn will fire.
KubeAPIDown is here to make sure that Prometheus and really any client can reach the kube-apiserver, which they can even when there is only one instance of kube-apiserver running. If they can't or that availability is disrupted, `KubeAPIErrorBudgetBurn` will fire.
Comment 23058588 by Marcel Härri at 2023-09-19T06:57:07.949+0000

Mar 12 18:52:24.937 - 58s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS
{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}

    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info"} >= 1
    [

        <*errors.errorString | 0xc0003fcf90>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.130.0.16:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"certified-operators\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-6f65df499f-d2dc7\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714138002.475,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.130.0.16:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"redhat-marketplace\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-6f65df499f-d2dc7\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714138002.475,\n      \"1\"\n    ]\n  }\n]",
        },

    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info"} >= 1
    [

        <*errors.errorString | 0xc001f036d0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.128.0.29:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"certified-operators\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-84dc4f6dbf-q4tfq\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714126975.535,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.128.0.29:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"redhat-marketplace\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-84dc4f6dbf-q4tfq\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714126975.535,\n      \"1\"\n    ]\n  }\n]",
        },

    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info"} >= 1
    [

        <*errors.errorString | 0xc0016f6950>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.128.0.29:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"certified-operators\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-84dc4f6dbf-q4tfq\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714129837.265,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.128.0.29:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"redhat-marketplace\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-84dc4f6dbf-q4tfq\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714129837.265,\n      \"1\"\n    ]\n  }\n]",
        },

    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info"} >= 1
    [

        <*errors.errorString | 0xc001e8ece0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.128.0.21:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"certified-operators\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-5658fc97f8-v82fp\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714097694.591,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.128.0.21:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"redhat-marketplace\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-5658fc97f8-v82fp\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714097694.591,\n      \"1\"\n    ]\n  },\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.128.0.21:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"redhat-operators\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-5658fc97f8-v82fp\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714097694.591,\n      \"1\"\n    ]\n  }\n]",
        },

    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info"} >= 1
    [

        <*errors.errorString | 0xc001c5dbe0>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.128.0.21:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"redhat-marketplace\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-5658fc97f8-v82fp\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1714100488.062,\n      \"1\"\n    ]\n  }\n]",
        },

    promQL query returned unexpected results:
    ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation",alertstate="firing",severity!="info"} >= 1
    [

        <*errors.errorString | 0xc001955930>{
            s: "promQL query returned unexpected results:\nALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards|KubeJobFailed|TelemeterClientFailures|Watchdog|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|KubePodNotReady|etcdMembersDown|etcdMembersDown|etcdGRPCRequestsSlow|etcdGRPCRequestsSlow|etcdHighNumberOfFailedGRPCRequests|etcdHighNumberOfFailedGRPCRequests|etcdMemberCommunicationSlow|etcdMemberCommunicationSlow|etcdNoLeader|etcdNoLeader|etcdHighFsyncDurations|etcdHighFsyncDurations|etcdHighCommitDurations|etcdHighCommitDurations|etcdInsufficientMembers|etcdInsufficientMembers|TargetDown|etcdHighNumberOfLeaderChanges|etcdHighNumberOfLeaderChanges|KubeAPIErrorBudgetBurn|KubeAPIErrorBudgetBurn|KubeClientErrors|KubeClientErrors|KubePersistentVolumeErrors|KubePersistentVolumeErrors|MCDDrainError|MCDDrainError|KubeMemoryOvercommit|KubeMemoryOvercommit|MCDPivotError|MCDPivotError|PrometheusOperatorWatchErrors|PrometheusOperatorWatchErrors|OVNKubernetesResourceRetryFailure|OVNKubernetesResourceRetryFailure|RedhatOperatorsCatalogError|RedhatOperatorsCatalogError|VSphereOpenshiftNodeHealthFail|VSphereOpenshiftNodeHealthFail|SamplesImagestreamImportFailing|SamplesImagestreamImportFailing|PodSecurityViolation\",alertstate=\"firing\",severity!=\"info\"} >= 1\n[\n  {\n    \"metric\": {\n      \"__name__\": \"ALERTS\",\n      \"alertname\": \"OperatorHubSourceError\",\n      \"alertstate\": \"firing\",\n      \"container\": \"catalog-operator\",\n      \"endpoint\": \"https-metrics\",\n      \"exported_namespace\": \"openshift-marketplace\",\n      \"instance\": \"10.130.0.27:8443\",\n      \"job\": \"catalog-operator-metrics\",\n      \"name\": \"certified-operators\",\n      \"namespace\": \"openshift-operator-lifecycle-manager\",\n      \"pod\": \"catalog-operator-5488d49f64-qfhcn\",\n      \"prometheus\": \"openshift-monitoring/k8s\",\n      \"service\": \"catalog-operator-metrics\",\n      \"severity\": \"warning\"\n    },\n    \"value\": [\n      1713946814.687,\n      \"1\"\n    ]\n  }\n]",
        },

# [sig-instrumentation][Late] OpenShift alerting rules [apigroup:image.openshift.io] should link to an HTTP(S) location if the runbook_url annotation is defined [Suite:openshift/conformance/parallel]
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:108]: Failed to fetch alerting rules: unable to parse response "{\"status\":\"success\",\"data\":{\"groups\":[{\"name\":\"CloudCredentialOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-59f29d20-14e3-45ac-95b2-817cbe434ddc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorDeprovisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsDeprovisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be cleaned up\",\"summary\":\"One or more CredentialsRequest CRs are unable to be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000117926,\"lastEvaluation\":\"2024-04-12T22:36:31.735304389Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorInsufficientCloudCreds\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"InsufficientCloudCreds\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.\",\"message\":\"Cluster's cloud credentials insufficient for minting or passthrough\",\"summary\":\"Problem with the available platform credentials.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000108965,\"lastEvaluation\":\"2024-04-12T22:36:31.735423516Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorProvisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsProvisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be fulfilled\",\"summary\":\"One or more CredentialsRequest CRs are unable to be processed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000098825,\"lastEvaluation\":\"2024-04-12T22:36:31.735204354Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorStaleCredentials\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"StaleCredentials\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.\",\"message\":\"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).\",\"summary\":\"One or more CredentialsRequest CRs are stale and should be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000105386,\"lastEvaluation\":\"2024-04-12T22:36:31.735533862Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorTargetNamespaceMissing\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"MissingTargetNamespace\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.\",\"message\":\"CredentialsRequest(s) pointing to non-existent namespace\",\"summary\":\"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000150643,\"lastEvaluation\":\"2024-04-12T22:36:31.735039966Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00053507,\"lastEvaluation\":\"2024-04-12T22:36:31.735037066Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-machine-approver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-02d4c4c8-9481-4482-a4eb-aa2196fb1abd.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineApproverMaxPendingCSRsReached\",\"query\":\"mapi_current_pending_csr \\u003e mapi_max_pending_csr\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of pending CertificateSigningRequests has exceeded the\\nmaximum threshold (current number of machine + 100). Check the\\npending CSRs to determine which machines need approval, also check\\nthat the nodelink controller is running in the openshift-machine-api\\nnamespace.\\n\",\"summary\":\"max pending CSRs threshold reached.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000264985,\"lastEvaluation\":\"2024-04-12T22:36:35.040054675Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000294385,\"lastEvaluation\":\"2024-04-12T22:36:35.039883304Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-tuning-operator.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-475016e5-fc0a-4af1-bd38-a45d26c2b85a.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NTODegraded\",\"query\":\"nto_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Node Tuning Operator is degraded. Review the \\\"node-tuning\\\" ClusterOperator object for further details.\",\"summary\":\"The Node Tuning Operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000087741,\"lastEvaluation\":\"2024-04-12T22:36:35.756300122Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NTOPodsNotReady\",\"query\":\"kube_pod_status_ready{condition=\\\"true\\\",namespace=\\\"openshift-cluster-node-tuning-operator\\\"} == 0\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.pod }} is not ready.\\nReview the \\\"Event\\\" objects in \\\"openshift-cluster-node-tuning-operator\\\" namespace for further details.\\n\",\"summary\":\"Pod {{ $labels.pod }} is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000291586,\"lastEvaluation\":\"2024-04-12T22:36:35.756006506Z\",\"type\":\"alerting\"},{\"name\":\"nto_custom_profiles:count\",\"query\":\"count by (_id) (nto_profile_calculated_total{profile!~\\\"openshift\\\",profile!~\\\"openshift-control-plane\\\",profile!~\\\"openshift-node\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000113402,\"lastEvaluation\":\"2024-04-12T22:36:35.756391654Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000503047,\"lastEvaluation\":\"2024-04-12T22:36:35.755518805Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"SamplesOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-5c7cc52f-a9cb-47fe-a845-20e9f9a26649.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SamplesDegraded\",\"query\":\"openshift_samples_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples could not be deployed and the operator is degraded. Review the \\\"openshift-samples\\\" ClusterOperator object for further details.\\n\",\"summary\":\"Samples operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000093025,\"lastEvaluation\":\"2024-04-12T22:36:47.104513835Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesImagestreamImportFailing\",\"query\":\"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \\u003e sum(openshift_samples_failed_imagestream_import_info)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports.  You can look at the \\\"openshift-samples\\\"\\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is detecting problems with imagestream image imports\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000264614,\"lastEvaluation\":\"2024-04-12T22:36:47.10424756Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesInvalidConfig\",\"query\":\"openshift_samples_invalidconfig_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator has been given an invalid configuration.\\n\",\"summary\":\"Samples operator Invalid configuration\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077034,\"lastEvaluation\":\"2024-04-12T22:36:47.10460794Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingSecret\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_secret\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator cannot find the samples pull secret in the openshift namespace.\\n\",\"summary\":\"Samples operator is not able to find secret\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000078804,\"lastEvaluation\":\"2024-04-12T22:36:47.104685694Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingTBRCredential\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_tbr_credential\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\\n\",\"summary\":\"Samples operator is not able to find the credentials for registry\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000072294,\"lastEvaluation\":\"2024-04-12T22:36:47.104765228Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesRetriesMissingOnImagestreamImportFailing\",\"query\":\"sum(openshift_samples_failed_imagestream_import_info) \\u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\\nimports are not occurring.  Contact support.  You can look at the \\\"openshift-samples\\\" ClusterOperator object\\nfor details. Most likely there are issues with the external image registry hosting the images that need to\\nbe investigated.  The list of ImageStreams that have failing imports are:\\n{{ range query \\\"openshift_samples_failed_imagestream_import_info \\u003e 0\\\" }}\\n  {{ .Labels.name }}\\n{{ end }}\\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is having problems with imagestream imports and its retries.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000430863,\"lastEvaluation\":\"2024-04-12T22:36:47.103814857Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesTBRInaccessibleOnBoot\",\"query\":\"openshift_samples_tbr_inaccessible_info == 1\",\"duration\":172800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"One of two situations has occurred.  Either\\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\\nassociated with each image stream tag from the samples catalog is\\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\\nassist the mirroring process.\\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\\n\",\"summary\":\"Samples operator is not able to access the registry on boot\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000079234,\"lastEvaluation\":\"2024-04-12T22:36:47.104838392Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001115289,\"lastEvaluation\":\"2024-04-12T22:36:47.103804797Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"default-storage-classes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MultipleDefaultStorageClasses\",\"query\":\"max_over_time(default_storage_class_count[5m]) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster storage operator monitors all storage classes configured in the cluster\\nand checks there is not more than one default StorageClass configured.\\n\",\"message\":\"StorageClass count check is failing (there should not be more than one default StorageClass)\",\"summary\":\"More than one default StorageClass detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000251395,\"lastEvaluation\":\"2024-04-12T22:36:37.477974261Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000378831,\"lastEvaluation\":\"2024-04-12T22:36:37.477148942Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodStartupStorageOperationsFailing\",\"query\":\"increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status!=\\\"success\\\"}[5m]) \\u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status=\\\"success\\\"}[5m]) == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Failing storage operation \\\"{{ $labels.operation_name }}\\\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\\nfrom starting for past 5 minutes.\\nPlease investigate Pods that are \\\"ContainerCreating\\\" on the node: \\\"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\\\".\\nEvents of the Pods should contain exact error message: \\\"oc describe pod -n \\u003cpod namespace\\u003e \\u003cpod name\\u003e\\\".\\n\",\"summary\":\"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003662945,\"lastEvaluation\":\"2024-04-12T22:36:49.473150763Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003676175,\"lastEvaluation\":\"2024-04-12T22:36:49.473141023Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-selinux.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"name\":\"cluster:volume_manager_selinux_pod_context_mismatch_total\",\"query\":\"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000381297,\"lastEvaluation\":\"2024-04-12T22:36:28.076271377Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volume_context_mismatch_errors_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000109212,\"lastEvaluation\":\"2024-04-12T22:36:28.076799817Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000142513,\"lastEvaluation\":\"2024-04-12T22:36:28.076655824Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volumes_admitted_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000286085,\"lastEvaluation\":\"2024-04-12T22:36:28.076910049Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001153521,\"lastEvaluation\":\"2024-04-12T22:36:28.075904055Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-operators\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-b7ec8a0e-ca37-48ff-91bd-48b44cd606d6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotEvaluateConditionalUpdates\",\"query\":\"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\\\"Recommended\\\",status=\\\"Unknown\\\"}) \\u003e= 3600)\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.\",\"summary\":\"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000149303,\"lastEvaluation\":\"2024-04-12T22:36:25.502645639Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterNotUpgradeable\",\"query\":\"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\\\"Upgradeable\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"Upgradeable\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0) (ne (len (label \\\"reason\\\" $value)) 0) }}{{label \\\"reason\\\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000282415,\"lastEvaluation\":\"2024-04-12T22:36:25.50115166Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDegraded\",\"query\":\"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\\\"Degraded\\\",job=\\\"cluster-version-operator\\\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\\\"cluster-version-operator\\\"})) == 1)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has been degraded for 30 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000644752,\"lastEvaluation\":\"2024-04-12T22:36:25.501743902Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDown\",\"query\":\"max by (namespace, name, reason) (cluster_operator_up{job=\\\"cluster-version-operator\\\"} == 0)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has not been available for 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000306316,\"lastEvaluation\":\"2024-04-12T22:36:25.501435936Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorFlapping\",\"query\":\"max by (namespace, name) (changes(cluster_operator_up{job=\\\"cluster-version-operator\\\"}[2m]) \\u003e 2)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator up status is changing often.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000254825,\"lastEvaluation\":\"2024-04-12T22:36:25.502389944Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001656412,\"lastEvaluation\":\"2024-04-12T22:36:25.50114171Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-b7ec8a0e-ca37-48ff-91bd-48b44cd606d6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotRetrieveUpdates\",\"query\":\"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \\u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\\\"RetrievedUpdates\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\",reason!=\\\"NoChannel\\\"}))\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"RetrievedUpdates\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0)}}{{label \\\"reason\\\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\\\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\\\n'}\\\"`{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000241142,\"lastEvaluation\":\"2024-04-12T22:36:39.739623191Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterReleaseNotAccepted\",\"query\":\"max by (namespace, name, reason) (cluster_operator_conditions{condition=\\\"ReleaseAccepted\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"The desired cluster release has not been accepted for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000125806,\"lastEvaluation\":\"2024-04-12T22:36:39.73996714Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterVersionOperatorDown\",\"query\":\"absent(up{job=\\\"cluster-version-operator\\\"} == 1)\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-cluster-version\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} For more information refer to {{ label \\\"url\\\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}\",\"summary\":\"Cluster version operator has disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000237812,\"lastEvaluation\":\"2024-04-12T22:36:39.739383739Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UpdateAvailable\",\"query\":\"sum by (channel, namespace, upstream) (cluster_version_available_updates) \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Your upstream update recommendation service recommends you update your cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000098335,\"lastEvaluation\":\"2024-04-12T22:36:39.739867784Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000646182,\"lastEvaluation\":\"2024-04-12T22:36:39.738991737Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift/console-operator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-13a53b5a-afd9-4211-9680-fe10e32351a7.yaml\",\"rules\":[{\"name\":\"cluster:console_auth_login_failures_total:sum\",\"query\":\"sum by (reason) (console_auth_login_failures_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000116322,\"lastEvaluation\":\"2024-04-12T22:36:29.799875967Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_requests_total:sum\",\"query\":\"sum(console_auth_login_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000191814,\"lastEvaluation\":\"2024-04-12T22:36:29.799544241Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_successes_total:sum\",\"query\":\"sum by (role) (console_auth_login_successes_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000133083,\"lastEvaluation\":\"2024-04-12T22:36:29.799738254Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_logout_requests_total:sum\",\"query\":\"sum by (reason) (console_auth_logout_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000080572,\"lastEvaluation\":\"2024-04-12T22:36:29.799993909Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_customization_perspectives_info:max\",\"query\":\"max by (name, state) (console_customization_perspectives_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000078672,\"lastEvaluation\":\"2024-04-12T22:36:29.800314685Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_plugins_info:max\",\"query\":\"max by (name, state) (console_plugins_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000163453,\"lastEvaluation\":\"2024-04-12T22:36:29.800149772Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_usage_users:max\",\"query\":\"max by (role) (console_usage_users)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000072861,\"lastEvaluation\":\"2024-04-12T22:36:29.800076061Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000865197,\"lastEvaluation\":\"2024-04-12T22:36:29.79953109Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-dns.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-be119138-53ca-458e-a33e-816e88c6ca38.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CoreDNSErrorsHigh\",\"query\":\"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\\\"SERVFAIL\\\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.\",\"summary\":\"CoreDNS serverfail\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000269855,\"lastEvaluation\":\"2024-04-12T22:36:51.98821851Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSHealthCheckSlow\",\"query\":\"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \\u003e 10\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})\",\"summary\":\"CoreDNS health checks\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000361857,\"lastEvaluation\":\"2024-04-12T22:36:51.987855743Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSPanicking\",\"query\":\"increase(coredns_panics_total[10m]) \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}\",\"summary\":\"CoreDNS panic\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000244395,\"lastEvaluation\":\"2024-04-12T22:36:51.987609808Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000887197,\"lastEvaluation\":\"2024-04-12T22:36:51.987603338Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"etcd\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-bdafb11d-bf7f-4480-ae4e-7cbeddeea773.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdDatabaseHighFragmentationRatio\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \\u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \\u003e 1.048576e+08\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.\",\"runbook_url\":\"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation\",\"summary\":\"etcd database size in use is less than 50% of the actual allocated storage.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000273924,\"lastEvaluation\":\"2024-04-12T22:36:44.199758574Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdDatabaseQuotaLowSpace\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \\u003e 95\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md\",\"summary\":\"etcd cluster database is running full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000176359,\"lastEvaluation\":\"2024-04-12T22:36:44.199396135Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdExcessiveDatabaseGrowth\",\"query\":\"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \\u003e etcd_server_quota_backend_bytes\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.\",\"summary\":\"etcd cluster database growing very fast.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.0001838,\"lastEvaluation\":\"2024-04-12T22:36:44.199573524Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighCommitDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.25\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile commit durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00036941,\"lastEvaluation\":\"2024-04-12T22:36:44.199025365Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000312917,\"lastEvaluation\":\"2024-04-12T22:36:44.198711528Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.5\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00038715,\"lastEvaluation\":\"2024-04-12T22:36:44.198323418Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedProposals\",\"query\":\"rate(etcd_server_proposals_failed_total{job=~\\\".*etcd.*\\\"}[15m]) \\u003e 5\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of proposal failures.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000140508,\"lastEvaluation\":\"2024-04-12T22:36:44.19818184Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMemberCommunicationSlow\",\"query\":\"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.15\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster member communication is slow.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000718188,\"lastEvaluation\":\"2024-04-12T22:36:44.197462382Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMembersDown\",\"query\":\"max without (endpoint) (sum without (instance) (up{job=~\\\".*etcd.*\\\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\\\".*etcd.*\\\"}[2m])) \\u003e 0.01)) \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": members are down ({{ $value }}).\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md\",\"summary\":\"etcd cluster members are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000461165,\"lastEvaluation\":\"2024-04-12T22:36:44.196853459Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdNoLeader\",\"query\":\"etcd_server_has_leader{job=~\\\".*etcd.*\\\"} == 0\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member {{ $labels.instance }} has no leader.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md\",\"summary\":\"etcd cluster has no leader.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000144798,\"lastEvaluation\":\"2024-04-12T22:36:44.197316204Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00318783,\"lastEvaluation\":\"2024-04-12T22:36:44.196846789Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-bdafb11d-bf7f-4480-ae4e-7cbeddeea773.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdGRPCRequestsSlow\",\"query\":\"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\\\"Defragment\\\",grpc_type=\\\"unary\\\",job=\\\"etcd\\\"}[10m]))) \\u003e 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md\",\"summary\":\"etcd grpc requests are slow\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.009185464,\"lastEvaluation\":\"2024-04-12T22:36:24.417977275Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 50\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.012922265,\"lastEvaluation\":\"2024-04-12T22:36:24.444522378Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 10\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.017354379,\"lastEvaluation\":\"2024-04-12T22:36:24.427166069Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfLeaderChanges\",\"query\":\"avg(changes(etcd_server_is_leader[10m])) \\u003e 5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.\",\"summary\":\"etcd cluster has high number of leader changes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000164563,\"lastEvaluation\":\"2024-04-12T22:36:24.457446793Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdInsufficientMembers\",\"query\":\"sum without (instance, pod) (up{job=\\\"etcd\\\"} == bool 1 and etcd_server_has_leader{job=\\\"etcd\\\"} == bool 1) \\u003c ((count without (instance, pod) (up{job=\\\"etcd\\\"}) + 1) / 2)\",\"duration\":180,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md\",\"summary\":\"etcd is reporting that a majority of instances are unavailable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000318496,\"lastEvaluation\":\"2024-04-12T22:36:24.457612596Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.037828601,\"lastEvaluation\":\"2024-04-12T22:36:24.417657431Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imageregistry.operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-2e046cbc-5e3f-419b-9e59-085c6eb3d543.yaml\",\"rules\":[{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.Create\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000205973,\"lastEvaluation\":\"2024-04-12T22:36:27.142541754Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.ServeBlob\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000348556,\"lastEvaluation\":\"2024-04-12T22:36:27.142191437Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Get\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000202683,\"lastEvaluation\":\"2024-04-12T22:36:27.142749038Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Put\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000142263,\"lastEvaluation\":\"2024-04-12T22:36:27.142952901Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001056526,\"lastEvaluation\":\"2024-04-12T22:36:27.141155686Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imagestreams.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-62089bf6-dfc3-4319-af63-38d6191b31ea.yaml\",\"rules\":[{\"name\":\"imageregistry:imagestreamtags_count:sum\",\"query\":\"sum by (location, source) (image_registry_image_stream_tags_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000297936,\"lastEvaluation\":\"2024-04-12T22:36:29.517918454Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000318487,\"lastEvaluation\":\"2024-04-12T22:36:29.517397281Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress-to-route-controller.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-dfd4b459-b669-489c-9020-22defa539ee2.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"IngressWithoutClassName\",\"query\":\"openshift_ingress_to_route_controller_ingress_without_class_name == 1\",\"duration\":86400,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"alerts\":[{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.130.0.46:8443\",\"job\":\"route-controller-manager\",\"name\":\"test\",\"namespace\":\"e2e-test-router-stress-g64j9\",\"pod\":\"route-controller-manager-5bdb9f48f4-nbc4h\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress test is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:21:56.656877779Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.00057098,\"lastEvaluation\":\"2024-04-12T22:36:26.658168642Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UnmanagedRoutes\",\"query\":\"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is a Route owned by an unmanaged Ingress.\",\"message\":\"Route {{ $labels.name }} is owned by an unmanaged Ingress.\",\"summary\":\"Route owned by an Ingress no longer managed\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00055756,\"lastEvaluation\":\"2024-04-12T22:36:26.658742192Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00108014,\"lastEvaluation\":\"2024-04-12T22:36:26.657113115Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-dfd4b459-b669-489c-9020-22defa539ee2.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HAProxyDown\",\"query\":\"haproxy_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"This alert fires when metrics report that HAProxy is down.\",\"message\":\"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}\",\"summary\":\"HAProxy is down\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000117152,\"lastEvaluation\":\"2024-04-12T22:36:36.029997778Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HAProxyReloadFail\",\"query\":\"template_router_reload_failure == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.\",\"message\":\"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes\",\"summary\":\"HAProxy reload failure\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000192924,\"lastEvaluation\":\"2024-04-12T22:36:36.029803404Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerDegraded\",\"query\":\"ingress_controller_conditions{condition=\\\"Degraded\\\"} == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController status is degraded.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\ndegraded: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is degraded\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000116962,\"lastEvaluation\":\"2024-04-12T22:36:36.0301163Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerUnavailable\",\"query\":\"ingress_controller_conditions{condition=\\\"Available\\\"} == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController is not available.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\nunavailable: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is unavailable\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000086392,\"lastEvaluation\":\"2024-04-12T22:36:36.030234183Z\",\"type\":\"alerting\"},{\"name\":\"cluster:openshift_route_info:tls_termination:sum\",\"query\":\"sum by (tls_termination) (openshift_route_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000177153,\"lastEvaluation\":\"2024-04-12T22:36:36.030713762Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:avg\",\"query\":\"avg(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000077671,\"lastEvaluation\":\"2024-04-12T22:36:36.030548629Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:max\",\"query\":\"max(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000088982,\"lastEvaluation\":\"2024-04-12T22:36:36.030458617Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:median\",\"query\":\"quantile(0.5, route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000085012,\"lastEvaluation\":\"2024-04-12T22:36:36.0306273Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:min\",\"query\":\"min(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000134313,\"lastEvaluation\":\"2024-04-12T22:36:36.030321674Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001099001,\"lastEvaluation\":\"2024-04-12T22:36:36.029794314Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"insights\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-9ee17981-1562-4044-be51-c623d1ec6f10.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"InsightsDisabled\",\"query\":\"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\\\"Disabled\\\",name=\\\"insights\\\"} == 1)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html\",\"summary\":\"Insights operator is disabled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000255843,\"lastEvaluation\":\"2024-04-12T22:36:26.980475055Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"InsightsRecommendationActive\",\"query\":\"insights_recommendation_active == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights recommendation \\\"{{ $labels.description }}\\\" with total risk \\\"{{ $labels.total_risk }}\\\" was detected on the cluster. More information is available at {{ $labels.info_link }}.\",\"summary\":\"An Insights recommendation is active for this cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077884,\"lastEvaluation\":\"2024-04-12T22:36:26.980878496Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SimpleContentAccessNotAvailable\",\"query\":\"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\\\"SCAAvailable\\\",name=\\\"insights\\\",reason=\\\"NotFound\\\"}[5m]) == 0)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.\",\"summary\":\"Simple content access certificates are not available.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000144768,\"lastEvaluation\":\"2024-04-12T22:36:26.980732438Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000520169,\"lastEvaluation\":\"2024-04-12T22:36:26.980225537Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pre-release-lifecycle\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-0d5178dd-4e23-49ca-884b-5d9c83f14fb4.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"APIRemovedInNextEUSReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.29\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next EUS version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.059891486,\"lastEvaluation\":\"2024-04-12T22:36:49.753094018Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"APIRemovedInNextReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.29\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.057387063,\"lastEvaluation\":\"2024-04-12T22:36:49.695703445Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.058073556,\"lastEvaluation\":\"2024-04-12T22:36:49.680137292Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-audit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-3528d707-66c1-425c-bbdb-459ae3d79e76.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AuditLogError\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\\\".+-apiserver\\\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\\\".+-apiserver\\\"}[5m])) \\u003e 0\",\"duration\":60,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"An API Server had an error writing to an audit log.\",\"summary\":\"An API Server instance was unable to write audit logs. This could be\\ntriggered by the node running out of space, or a malicious actor\\ntampering with the audit logs.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000601002,\"lastEvaluation\":\"2024-04-12T22:36:35.38840949Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000720889,\"lastEvaluation\":\"2024-04-12T22:36:35.388292604Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"control-plane-cpu-utilization\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-7d5403fb-3bd5-4f24-ada6-427ad465fbde.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000343136,\"lastEvaluation\":\"2024-04-12T22:36:36.635727642Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000291696,\"lastEvaluation\":\"2024-04-12T22:36:36.635435026Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneCPU\",\"query\":\"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / count(kube_node_role{role=\\\"master\\\"}) \\u003e 60\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000682083,\"lastEvaluation\":\"2024-04-12T22:36:36.634751593Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001586624,\"lastEvaluation\":\"2024-04-12T22:36:36.634190473Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"api-performance\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-60c107f0-5c52-40c2-b772-4473cfac1e20.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000199404,\"lastEvaluation\":\"2024-04-12T22:36:23.675383044Z\",\"type\":\"recording\"},{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000169833,\"lastEvaluation\":\"2024-04-12T22:36:23.675583668Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.023276721,\"lastEvaluation\":\"2024-04-12T22:36:23.481059232Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.026017973,\"lastEvaluation\":\"2024-04-12T22:36:23.504341813Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:1m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000291196,\"lastEvaluation\":\"2024-04-12T22:36:23.480562282Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:5m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000202873,\"lastEvaluation\":\"2024-04-12T22:36:23.480855048Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003580368,\"lastEvaluation\":\"2024-04-12T22:36:23.654924816Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003470766,\"lastEvaluation\":\"2024-04-12T22:36:23.658508734Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00057902,\"lastEvaluation\":\"2024-04-12T22:36:23.647286082Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000487939,\"lastEvaluation\":\"2024-04-12T22:36:23.64826652Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001961498,\"lastEvaluation\":\"2024-04-12T22:36:23.643433938Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001887095,\"lastEvaluation\":\"2024-04-12T22:36:23.645397016Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004412413,\"lastEvaluation\":\"2024-04-12T22:36:23.614417119Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004283691,\"lastEvaluation\":\"2024-04-12T22:36:23.618832142Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009910747,\"lastEvaluation\":\"2024-04-12T22:36:23.623383119Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010133423,\"lastEvaluation\":\"2024-04-12T22:36:23.633297177Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000130022,\"lastEvaluation\":\"2024-04-12T22:36:23.623118444Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000131973,\"lastEvaluation\":\"2024-04-12T22:36:23.623249546Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002740862,\"lastEvaluation\":\"2024-04-12T22:36:23.609015826Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002656361,\"lastEvaluation\":\"2024-04-12T22:36:23.611758628Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002852824,\"lastEvaluation\":\"2024-04-12T22:36:23.667102767Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002514648,\"lastEvaluation\":\"2024-04-12T22:36:23.669957711Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002680531,\"lastEvaluation\":\"2024-04-12T22:36:23.66198185Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002435906,\"lastEvaluation\":\"2024-04-12T22:36:23.664664841Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002906495,\"lastEvaluation\":\"2024-04-12T22:36:23.672474479Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000215944,\"lastEvaluation\":\"2024-04-12T22:36:23.480176795Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000166313,\"lastEvaluation\":\"2024-04-12T22:36:23.480393979Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00211391,\"lastEvaluation\":\"2024-04-12T22:36:23.576715704Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.022041597,\"lastEvaluation\":\"2024-04-12T22:36:23.530366036Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.021222472,\"lastEvaluation\":\"2024-04-12T22:36:23.552412404Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007422281,\"lastEvaluation\":\"2024-04-12T22:36:23.57913066Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007539133,\"lastEvaluation\":\"2024-04-12T22:36:23.586555421Z\",\"type\":\"recording\"},{\"name\":\"instance_request_kind:apiserver_current_inflight_requests:sum\",\"query\":\"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000296876,\"lastEvaluation\":\"2024-04-12T22:36:23.578832524Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.091631305,\"lastEvaluation\":\"2024-04-12T22:36:51.904027503Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.102604299,\"lastEvaluation\":\"2024-04-12T22:36:51.996960149Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.322721545,\"lastEvaluation\":\"2024-04-12T22:36:22.716943042Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.348291371,\"lastEvaluation\":\"2024-04-12T22:36:23.039670917Z\",\"type\":\"recording\"},{\"name\":\"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum\",\"query\":\"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000398517,\"lastEvaluation\":\"2024-04-12T22:36:23.647866783Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009196814,\"lastEvaluation\":\"2024-04-12T22:36:23.443928018Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007984112,\"lastEvaluation\":\"2024-04-12T22:36:23.453128202Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003067948,\"lastEvaluation\":\"2024-04-12T22:36:23.573644906Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_storage_objects:max\",\"query\":\"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002858074,\"lastEvaluation\":\"2024-04-12T22:36:23.675755251Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000373647,\"lastEvaluation\":\"2024-04-12T22:36:23.654257964Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000288945,\"lastEvaluation\":\"2024-04-12T22:36:23.654634331Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.283265351,\"lastEvaluation\":\"2024-04-12T22:36:51.386615443Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.234128367,\"lastEvaluation\":\"2024-04-12T22:36:51.669889875Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.028779516,\"lastEvaluation\":\"2024-04-12T22:36:23.387969837Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.027162865,\"lastEvaluation\":\"2024-04-12T22:36:23.416757473Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007143945,\"lastEvaluation\":\"2024-04-12T22:36:23.594097204Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007768227,\"lastEvaluation\":\"2024-04-12T22:36:23.601244029Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002966196,\"lastEvaluation\":\"2024-04-12T22:36:23.648756859Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002529138,\"lastEvaluation\":\"2024-04-12T22:36:23.651726346Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.202137845,\"lastEvaluation\":\"2024-04-12T22:36:52.287428993Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.213206304,\"lastEvaluation\":\"2024-04-12T22:36:52.512496584Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.090115927,\"lastEvaluation\":\"2024-04-12T22:36:52.099571859Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.093912566,\"lastEvaluation\":\"2024-04-12T22:36:52.193510476Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011175321,\"lastEvaluation\":\"2024-04-12T22:36:23.461114464Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007879309,\"lastEvaluation\":\"2024-04-12T22:36:23.472293786Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":2.215395111,\"lastEvaluation\":\"2024-04-12T22:36:21.386081653Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-requests-in-flight\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-6824c540-c854-48f2-9d56-c5eabf4a2b2f.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m\",\"query\":\"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver\\\"})[2m:])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000474705,\"lastEvaluation\":\"2024-04-12T22:36:27.870854345Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000492436,\"lastEvaluation\":\"2024-04-12T22:36:27.870839924Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-basic\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4f08eea7-5a77-4739-ba5f-4ebfa6a5c384.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1h) \\u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \\u003e (14.4 * 0.01)\",\"duration\":120,\"labels\":{\"long\":\"1h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"5m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000372347,\"lastEvaluation\":\"2024-04-12T22:36:25.801155645Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate6h) \\u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \\u003e (6 * 0.01)\",\"duration\":900,\"labels\":{\"long\":\"6h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"30m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000186373,\"lastEvaluation\":\"2024-04-12T22:36:25.801529542Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000451685,\"lastEvaluation\":\"2024-04-12T22:36:25.801132033Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4f08eea7-5a77-4739-ba5f-4ebfa6a5c384.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.015164194,\"lastEvaluation\":\"2024-04-12T22:36:38.558555321Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.010181167,\"lastEvaluation\":\"2024-04-12T22:36:38.588315118Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.009728989,\"lastEvaluation\":\"2024-04-12T22:36:38.548823922Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.007657447,\"lastEvaluation\":\"2024-04-12T22:36:38.598549263Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.010208408,\"lastEvaluation\":\"2024-04-12T22:36:38.538611644Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.005851462,\"lastEvaluation\":\"2024-04-12T22:36:38.606210031Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.014587073,\"lastEvaluation\":\"2024-04-12T22:36:38.573724735Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.009160747,\"lastEvaluation\":\"2024-04-12T22:36:38.612063442Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.121687879,\"lastEvaluation\":\"2024-04-12T22:36:39.048674253Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.130384218,\"lastEvaluation\":\"2024-04-12T22:36:38.918282635Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.123482134,\"lastEvaluation\":\"2024-04-12T22:36:38.794796241Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.083649567,\"lastEvaluation\":\"2024-04-12T22:36:38.637102042Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.073791771,\"lastEvaluation\":\"2024-04-12T22:36:38.72098503Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.008557175,\"lastEvaluation\":\"2024-04-12T22:36:38.621226979Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.007312438,\"lastEvaluation\":\"2024-04-12T22:36:38.629786694Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.631766988,\"lastEvaluation\":\"2024-04-12T22:36:38.538600094Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-extended\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-168fb4ca-48ed-4fa6-95a7-20c6c46562ef.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1d) \\u003e (3 * 0.01) and sum(apiserver_request:burnrate2h) \\u003e (3 * 0.01)\",\"duration\":3600,\"labels\":{\"long\":\"1d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"2h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000321587,\"lastEvaluation\":\"2024-04-12T22:36:35.100389895Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate3d) \\u003e (1 * 0.01) and sum(apiserver_request:burnrate6h) \\u003e (1 * 0.01)\",\"duration\":10800,\"labels\":{\"long\":\"3d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"6h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000424182,\"lastEvaluation\":\"2024-04-12T22:36:35.10077552Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001123449,\"lastEvaluation\":\"2024-04-12T22:36:35.100083524Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-168fb4ca-48ed-4fa6-95a7-20c6c46562ef.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.019031292,\"lastEvaluation\":\"2024-04-12T22:36:31.518244181Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.010686038,\"lastEvaluation\":\"2024-04-12T22:36:31.553264243Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.022732989,\"lastEvaluation\":\"2024-04-12T22:36:31.495505362Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.012240721,\"lastEvaluation\":\"2024-04-12T22:36:31.563953511Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.015979289,\"lastEvaluation\":\"2024-04-12T22:36:31.537281113Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.011007505,\"lastEvaluation\":\"2024-04-12T22:36:31.576198032Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.091709845,\"lastEvaluation\":\"2024-04-12T22:36:31.495498572Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-5005c833-ea1b-4666-ae08-3bea362dc2d3.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TechPreviewNoUpgrade\",\"query\":\"cluster_feature_set{name!=\\\"\\\",namespace=\\\"openshift-kube-apiserver-operator\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.\",\"summary\":\"Cluster has enabled tech preview features that will prevent upgrades.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002395806,\"lastEvaluation\":\"2024-04-12T22:36:45.48300113Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002119112,\"lastEvaluation\":\"2024-04-12T22:36:45.482834302Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pod-security-violation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-be42b924-fa88-4cdc-b097-1e5e3015f2ef.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace!=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created in namespace \\\"{{ $labels.ocp_namespace }}\\\" but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000136227,\"lastEvaluation\":\"2024-04-12T22:36:27.163648332Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads users created in the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000253205,\"lastEvaluation\":\"2024-04-12T22:36:27.163347548Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000380587,\"lastEvaluation\":\"2024-04-12T22:36:27.163343618Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-59a05266-d32f-408a-b2c9-134a0aa47a1f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"GarbageCollectorSyncFailed\",\"query\":\"rate(garbagecollector_controller_resources_sync_error_total[5m]) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md\",\"summary\":\"There was a problem with syncing the resources for garbage collection.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000191034,\"lastEvaluation\":\"2024-04-12T22:36:26.687631783Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeControllerManagerDown\",\"query\":\"absent(up{job=\\\"kube-controller-manager\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-controller-manager\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeControllerManager has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000311836,\"lastEvaluation\":\"2024-04-12T22:36:26.686338099Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetAtLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \\u003e 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md\",\"summary\":\"The pod disruption budget is preventing further disruption to pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000466764,\"lastEvaluation\":\"2024-04-12T22:36:26.686652636Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \\u003c kube_poddisruptionbudget_status_desired_healthy)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md\",\"summary\":\"The pod disruption budget registers insufficient amount of pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000365197,\"lastEvaluation\":\"2024-04-12T22:36:26.687264646Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00158003,\"lastEvaluation\":\"2024-04-12T22:36:26.686246467Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-14fb49f3-f1fb-4874-93f5-bc12c6437bcf.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeSchedulerDown\",\"query\":\"absent(up{job=\\\"scheduler\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-scheduler\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeScheduler has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000321926,\"lastEvaluation\":\"2024-04-12T22:36:31.339316817Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000336017,\"lastEvaluation\":\"2024-04-12T22:36:31.339306417Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"scheduler-legacy-policy-deprecated\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-14fb49f3-f1fb-4874-93f5-bc12c6437bcf.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SchedulerLegacyPolicySet\",\"query\":\"cluster_legacy_scheduler_policy \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.\",\"summary\":\"Legacy scheduler policy API in use by the scheduler.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000216224,\"lastEvaluation\":\"2024-04-12T22:36:43.443324055Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000229705,\"lastEvaluation\":\"2024-04-12T22:36:43.443314645Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-api-operator-metrics-collector-up\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineAPIOperatorMetricsCollectionFailing\",\"query\":\"mapi_mao_collector_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"For more details:  oc logs \\u003cmachine-api-operator-pod-name\\u003e -n openshift-machine-api\",\"summary\":\"machine api operator metrics collection is failing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000214694,\"lastEvaluation\":\"2024-04-12T22:36:23.972886041Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000226154,\"lastEvaluation\":\"2024-04-12T22:36:23.972879051Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-health-check-unterminated-short-circuit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineHealthCheckUnterminatedShortCircuit\",\"query\":\"mapi_machinehealthcheck_short_circuit == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\\nthe status of machines in the cluster.\\n\",\"summary\":\"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000221112,\"lastEvaluation\":\"2024-04-12T22:36:51.928398519Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000233643,\"lastEvaluation\":\"2024-04-12T22:36:51.928389159Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-not-yet-deleted\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineNotYetDeleted\",\"query\":\"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\\\"Deleting\\\"}[15m])) \\u003e 0\",\"duration\":21600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine is not properly deleting, this may be due to a configuration issue with the\\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\\nlong termination periods which are preventing deletion.\\n\",\"summary\":\"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000272524,\"lastEvaluation\":\"2024-04-12T22:36:44.941617306Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000285856,\"lastEvaluation\":\"2024-04-12T22:36:44.941608115Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-with-no-running-phase\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithNoRunningPhase\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\\\"Running|Deleting\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine has been without a Running or Deleting phase for more than 60 minutes.\\nThe machine may not have been provisioned properly from the infrastructure provider, or\\nit might have issues with CertificateSigningRequests being approved.\\n\",\"summary\":\"machine {{ $labels.name }} is in phase: {{ $labels.phase }}\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000326337,\"lastEvaluation\":\"2024-04-12T22:36:31.397113273Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000264484,\"lastEvaluation\":\"2024-04-12T22:36:31.396641077Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-without-valid-node-ref\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithoutValidNode\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"If the machine never became a node, you should diagnose the machine related failures.\\nIf the node was deleted from the API, you may delete the machine if appropriate.\\n\",\"summary\":\"machine {{ $labels.name }} does not have valid node reference\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000335148,\"lastEvaluation\":\"2024-04-12T22:36:45.843108201Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000363587,\"lastEvaluation\":\"2024-04-12T22:36:45.836234369Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-drain-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCDrainError\",\"query\":\"mcc_drain_err \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000240684,\"lastEvaluation\":\"2024-04-12T22:36:26.126555741Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000254695,\"lastEvaluation\":\"2024-04-12T22:36:26.126545581Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-pool-alert\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCPoolAlert\",\"query\":\"mcc_pool_alert \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000245005,\"lastEvaluation\":\"2024-04-12T22:36:45.350111834Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000257436,\"lastEvaluation\":\"2024-04-12T22:36:45.350103014Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"os-image-override.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"name\":\"os_image_url_override:sum\",\"query\":\"sum(os_image_url_override)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000269855,\"lastEvaluation\":\"2024-04-12T22:36:25.675348321Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000312607,\"lastEvaluation\":\"2024-04-12T22:36:25.675173108Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"extremely-high-individual-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneMemory\",\"query\":\"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 90\",\"duration\":2700,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.\",\"summary\":\"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000650663,\"lastEvaluation\":\"2024-04-12T22:36:39.133743062Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000675846,\"lastEvaluation\":\"2024-04-12T22:36:39.132725259Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"high-overall-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneMemory\",\"query\":\"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 60\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.\",\"summary\":\"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000830794,\"lastEvaluation\":\"2024-04-12T22:36:37.080318668Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000622492,\"lastEvaluation\":\"2024-04-12T22:36:37.079547936Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-kubelet-health-state-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeletHealthState\",\"query\":\"mcd_kubelet_state \\u003e 2\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet health failure threshold reached\",\"summary\":\"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000346499,\"lastEvaluation\":\"2024-04-12T22:36:36.713249675Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000241255,\"lastEvaluation\":\"2024-04-12T22:36:36.712959868Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-pivot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDPivotError\",\"query\":\"mcd_pivot_errors_total \\u003e 0\",\"duration\":120,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000243424,\"lastEvaluation\":\"2024-04-12T22:36:26.054822042Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000256836,\"lastEvaluation\":\"2024-04-12T22:36:26.054811371Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-reboot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDRebootError\",\"query\":\"mcd_reboots_failed_total \\u003e 0\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000279045,\"lastEvaluation\":\"2024-04-12T22:36:24.37248846Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000271625,\"lastEvaluation\":\"2024-04-12T22:36:24.3717978Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"system-memory-exceeds-reservation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SystemMemoryExceedsReservation\",\"query\":\"sum by (node) (container_memory_rss{id=\\\"/system.slice\\\"}) \\u003e ((sum by (node) (kube_node_status_capacity{resource=\\\"memory\\\"} - kube_node_status_allocatable{resource=\\\"memory\\\"})) * 0.95)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).\",\"summary\":\"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000574741,\"lastEvaluation\":\"2024-04-12T22:36:44.345946605Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000548709,\"lastEvaluation\":\"2024-04-12T22:36:44.345632125Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"operator.marketplace.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-marketplace-marketplace-alert-rules-714df196-fc7d-484a-8308-076cc71239c1.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"OperatorHubSourceError\",\"query\":\"catalogsource_ready{exported_namespace=\\\"openshift-marketplace\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Operators shipped via the {{ $labels.name }} source are not available for installation until the issue is fixed. Operators already installed from this source will not receive updates until issue is fixed. Inspect the status of the pod owned by {{ $labels.name }} source in the openshift-marketplace namespace (oc -n openshift-marketplace get pods -l olm.catalogSource={{ $labels.name }}) to diagnose and repair.\",\"summary\":\"The {{ $labels.name }} source is in non-ready state for more than 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000205474,\"lastEvaluation\":\"2024-04-12T22:36:26.137690192Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000213214,\"lastEvaluation\":\"2024-04-12T22:36:26.137684882Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"alertmanager.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-alertmanager-main-rules-b89ef7fc-8e25-4a61-90d5-f3ea1f8d7983.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterDown\",\"query\":\"(count by (namespace, service) (avg_over_time(up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c 0.5) / count by (namespace, service) (up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) \\u003e= 0.5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of Alertmanager instances within the {{$labels.job}} cluster have been up for less than half of the last 5m.\",\"summary\":\"Half or more of the Alertmanager instances within the same cluster are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000275936,\"lastEvaluation\":\"2024-04-12T22:36:40.32919809Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterFailedToSendAlerts\",\"query\":\"min by (namespace, service, integration) (rate(alertmanager_notifications_failed_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The minimum notification failure rate to {{ $labels.integration }} sent from any instance in the {{$labels.job}} cluster is {{ $value | humanizePercentage }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerClusterFailedToSendAlerts.md\",\"summary\":\"All Alertmanager instances in a cluster failed to send notifications to a critical integration.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000824656,\"lastEvaluation\":\"2024-04-12T22:36:40.328170879Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerConfigInconsistent\",\"query\":\"count by (namespace, service) (count_values by (namespace, service) (\\\"config_hash\\\", alertmanager_config_hash{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) != 1\",\"duration\":1200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager instances within the {{$labels.job}} cluster have different configurations.\",\"summary\":\"Alertmanager instances within the same cluster have different configurations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000199453,\"lastEvaluation\":\"2024-04-12T22:36:40.328996846Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedReload\",\"query\":\"max_over_time(alertmanager_config_last_reload_successful{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Configuration has failed to load for {{ $labels.namespace }}/{{ $labels.pod}}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedReload.md\",\"summary\":\"Reloading an Alertmanager configuration has failed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000355179,\"lastEvaluation\":\"2024-04-12T22:36:40.326475027Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedToSendAlerts\",\"query\":\"(rate(alertmanager_notifications_failed_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} failed to send {{ $value | humanizePercentage }} of notifications to {{ $labels.integration }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedToSendAlerts.md\",\"summary\":\"An Alertmanager instance failed to send notifications.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000700037,\"lastEvaluation\":\"2024-04-12T22:36:40.327184894Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerMembersInconsistent\",\"query\":\"max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c on (namespace, service) group_left () count by (namespace, service) (max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]))\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} has only found {{ $value }} members of the {{$labels.job}} cluster.\",\"summary\":\"A member of an Alertmanager cluster has not found all other cluster members.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000351658,\"lastEvaluation\":\"2024-04-12T22:36:40.326831955Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003114101,\"lastEvaluation\":\"2024-04-12T22:36:40.326362195Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-list-watch.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"apiserver_list_watch_request_success_total:rate:sum\",\"query\":\"sum by (verb) (rate(apiserver_request_total{code=~\\\"2..\\\",verb=~\\\"LIST|WATCH\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011091991,\"lastEvaluation\":\"2024-04-12T22:36:49.03612487Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.01343978,\"lastEvaluation\":\"2024-04-12T22:36:49.035981344Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"Watchdog\",\"query\":\"vector(1)\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"alerts\":[{\"labels\":{\"alertname\":\"Watchdog\",\"namespace\":\"openshift-monitoring\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"state\":\"firing\",\"activeAt\":\"2024-04-12T21:38:00.04805628Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.00036572,\"lastEvaluation\":\"2024-04-12T22:36:30.049373349Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000325186,\"lastEvaluation\":\"2024-04-12T22:36:30.049005388Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"count:up0\",\"query\":\"count without (instance, pod, node) (up == 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000584401,\"lastEvaluation\":\"2024-04-12T22:36:30.562035441Z\",\"type\":\"recording\"},{\"name\":\"count:up1\",\"query\":\"count without (instance, pod, node) (up == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001188013,\"lastEvaluation\":\"2024-04-12T22:36:30.560845548Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001786285,\"lastEvaluation\":\"2024-04-12T22:36:30.560835887Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-node-recording.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:node_cpu:ratio\",\"query\":\"cluster:node_cpu:sum_rate5m / count(sum by (instance, cpu) (node_cpu_seconds_total))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001001909,\"lastEvaluation\":\"2024-04-12T22:36:31.717688757Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:sum_rate5m\",\"query\":\"sum(rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000501319,\"lastEvaluation\":\"2024-04-12T22:36:31.717186508Z\",\"type\":\"recording\"},{\"name\":\"instance:node_cpu:rate:sum\",\"query\":\"sum by (instance) (rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000744524,\"lastEvaluation\":\"2024-04-12T22:36:31.715888793Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_receive_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000298946,\"lastEvaluation\":\"2024-04-12T22:36:31.716635227Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_transmit_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_transmit_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000250275,\"lastEvaluation\":\"2024-04-12T22:36:31.716935283Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002811523,\"lastEvaluation\":\"2024-04-12T22:36:31.715881853Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-recurring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:usage:workload:capacity_physical_cpu_core_seconds\",\"query\":\"sum_over_time(workload:capacity_physical_cpu_cores:sum[30s:1s]) + ((cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) or (absent(cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) * 0))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000427389,\"lastEvaluation\":\"2024-04-12T22:36:34.377895604Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000439449,\"lastEvaluation\":\"2024-04-12T22:36:34.377886824Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-network\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeNetworkInterfaceFlapping\",\"query\":\"changes(node_network_up{device!~\\\"veth.+|tunbr\\\",job=\\\"node-exporter\\\"}[2m]) \\u003e 2\",\"duration\":120,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Network interface \\\"{{ $labels.device }}\\\" changing its up status often on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\",\"summary\":\"Network interface is often changing its status\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000479155,\"lastEvaluation\":\"2024-04-12T22:36:25.834301237Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000651173,\"lastEvaluation\":\"2024-04-12T22:36:25.833252363Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-build.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"openshift:build_by_strategy:sum\",\"query\":\"sum by (strategy) (openshift_build_status_phase_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000352997,\"lastEvaluation\":\"2024-04-12T22:36:39.546313771Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000366027,\"lastEvaluation\":\"2024-04-12T22:36:39.546304101Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd-telemetry.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000471499,\"lastEvaluation\":\"2024-04-12T22:36:46.87614077Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000474385,\"lastEvaluation\":\"2024-04-12T22:36:46.874714269Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000240762,\"lastEvaluation\":\"2024-04-12T22:36:46.874471787Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_use_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000215515,\"lastEvaluation\":\"2024-04-12T22:36:46.875923125Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000687606,\"lastEvaluation\":\"2024-04-12T22:36:46.875190305Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002345756,\"lastEvaluation\":\"2024-04-12T22:36:46.874268973Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TargetDown\",\"query\":\"100 * ((1 - sum by (job, namespace, service) (up and on (namespace, pod) kube_pod_info) / count by (job, namespace, service) (up and on (namespace, pod) kube_pod_info)) or (count by (job, namespace, service) (up == 0) / count by (job, namespace, service) (up))) \\u003e 10\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ printf \\\"%.4g\\\" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.\",\"summary\":\"Some targets were not reachable from the monitoring server for an extended period of time.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00725452,\"lastEvaluation\":\"2024-04-12T22:36:49.131392564Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00726747,\"lastEvaluation\":\"2024-04-12T22:36:49.131383473Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:ingress_controller_aws_nlb_active:sum\",\"query\":\"sum(ingress_controller_aws_nlb_active) or vector(0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000117253,\"lastEvaluation\":\"2024-04-12T22:36:47.632569255Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_in:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_in_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000150232,\"lastEvaluation\":\"2024-04-12T22:36:47.628183781Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_out:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_out_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000159203,\"lastEvaluation\":\"2024-04-12T22:36:47.628335493Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_connections:sum\",\"query\":\"sum(haproxy_frontend_current_sessions)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000108811,\"lastEvaluation\":\"2024-04-12T22:36:47.628495827Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace=~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00157298,\"lastEvaluation\":\"2024-04-12T22:36:47.629900964Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001092682,\"lastEvaluation\":\"2024-04-12T22:36:47.631475284Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace!~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000785795,\"lastEvaluation\":\"2024-04-12T22:36:47.628605789Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00050599,\"lastEvaluation\":\"2024-04-12T22:36:47.629393424Z\",\"type\":\"recording\"},{\"name\":\"code:cluster:ingress_http_request_count:rate5m:sum\",\"query\":\"sum by (code) (rate(haproxy_server_http_responses_total[5m]) \\u003e 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001585241,\"lastEvaluation\":\"2024-04-12T22:36:47.62659673Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006102148,\"lastEvaluation\":\"2024-04-12T22:36:47.62658617Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-kubernetes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"AlertmanagerReceiversNotConfigured\",\"query\":\"cluster:alertmanager_integrations:max == 0\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"alerts\":[{\"labels\":{\"alertname\":\"AlertmanagerReceiversNotConfigured\",\"namespace\":\"openshift-monitoring\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"state\":\"firing\",\"activeAt\":\"2024-04-12T21:39:17.894182481Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000165894,\"lastEvaluation\":\"2024-04-12T22:36:47.996157584Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterMonitoringOperatorReconciliationErrors\",\"query\":\"max_over_time(cluster_monitoring_operator_last_reconciliation_successful[5m]) == 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors are occurring during reconciliation cycles. Inspect the cluster-monitoring-operator log for potential root causes.\",\"summary\":\"Cluster Monitoring Operator is experiencing unexpected reconciliation errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077241,\"lastEvaluation\":\"2024-04-12T22:36:47.996079603Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentReplicasMismatch\",\"query\":\"(((kube_deployment_spec_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_deployment_status_replicas_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_deployment_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)) * on () group_left () cluster:control_plane:all_nodes_ready) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes. This indicates that cluster infrastructure is unable to start or restart the necessary components. This most often occurs when one or more nodes are down or partioned from the cluster, or a fault occurs on the node that prevents the workload from starting. In rare cases this may indicate a new version of a cluster component cannot start due to a bug or configuration error. Assess the pods for this deployment to verify they are running on healthy nodes and then contact support.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeDeploymentReplicasMismatch.md\",\"summary\":\"Deployment has not matched the expected number of replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002321535,\"lastEvaluation\":\"2024-04-12T22:36:47.996324498Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodNotScheduled\",\"query\":\"last_over_time(kube_pod_status_unschedulable{namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} cannot be scheduled for more than 30 minutes.\\nCheck the details of the pod with the following command:\\noc describe -n {{ $labels.namespace }} pod {{ $labels.pod }}\",\"summary\":\"Pod cannot be scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000541001,\"lastEvaluation\":\"2024-04-12T22:36:48.025027584Z\",\"type\":\"alerting\"},{\"name\":\"cluster:alertmanager_integrations:max\",\"query\":\"max(alertmanager_integrations{namespace=\\\"openshift-monitoring\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000121752,\"lastEvaluation\":\"2024-04-12T22:36:47.995154745Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (label_replace(cluster:infra_nodes, \\\"label_node_role_kubernetes_io\\\", \\\"infra\\\", \\\"\\\", \\\"\\\") * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000464569,\"lastEvaluation\":\"2024-04-12T22:36:47.964889868Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000130143,\"lastEvaluation\":\"2024-04-12T22:36:47.966423848Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_sockets_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (max by (node, package, label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000144733,\"lastEvaluation\":\"2024-04-12T22:36:47.994840189Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_memory_bytes:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000365347,\"lastEvaluation\":\"2024-04-12T22:36:47.966555301Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_cpu_usage:ratio\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m])) / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008544355,\"lastEvaluation\":\"2024-04-12T22:36:47.952716206Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_spec_cpu_shares:ratio\",\"query\":\"sum(container_spec_cpu_shares{container=\\\"\\\",pod!=\\\"\\\"}) / 1000 / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008869612,\"lastEvaluation\":\"2024-04-12T22:36:47.943845054Z\",\"type\":\"recording\"},{\"name\":\"cluster:control_plane:all_nodes_ready\",\"query\":\"sum(min by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}) and max by (node) (kube_node_role{role=\\\"master\\\"})) == bool sum(kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000271916,\"lastEvaluation\":\"2024-04-12T22:36:47.995738736Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_hyperthreading\",\"query\":\"clamp_max(label_replace(sum by (instance, package, core) (node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_replace(label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003e 1, \\\"label_node_hyperthread_enabled\\\", \\\"true\\\", \\\"instance\\\", \\\"(.*)\\\") or on (instance, package) label_replace(sum by (instance, package, core) (label_replace(node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003c= 1, \\\"label_node_hyperthread_enabled\\\", \\\"false\\\", \\\"instance\\\", \\\"(.*)\\\"), 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000775525,\"lastEvaluation\":\"2024-04-12T22:36:47.965356087Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_node_labels\",\"query\":\"topk by (node) (1, cluster:nodes_roles) * on (node) group_right (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_node_openshift_io_os_id, label_kubernetes_io_arch, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) label_replace(cluster:cpu_core_hyperthreading, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000288696,\"lastEvaluation\":\"2024-04-12T22:36:47.966133642Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_usage_cores:sum\",\"query\":\"sum(1 - rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[2m]) * on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{pod=~\\\"node-exporter.+\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00054354,\"lastEvaluation\":\"2024-04-12T22:36:47.966921808Z\",\"type\":\"recording\"},{\"name\":\"cluster:hyperthread_enabled_nodes\",\"query\":\"kube_node_labels and on (node) (sum by (node, package, core) (label_replace(node_cpu_info, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")) == 2)\",\"labels\":{\"label_node_hyperthread_enabled\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000349047,\"lastEvaluation\":\"2024-04-12T22:36:47.964357608Z\",\"type\":\"recording\"},{\"name\":\"cluster:infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"infra\\\"})\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000170604,\"lastEvaluation\":\"2024-04-12T22:36:47.963852708Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolume_plugin_type_counts:sum\",\"query\":\"sum by (plugin_name, volume_mode) (pv_collector_total_pv_count{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000459179,\"lastEvaluation\":\"2024-04-12T22:36:47.995278127Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_resource_requests_storage_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000755684,\"lastEvaluation\":\"2024-04-12T22:36:47.985956067Z\",\"type\":\"recording\"},{\"name\":\"cluster:kubelet_volume_stats_used_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kubelet_volume_stats_used_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000401228,\"lastEvaluation\":\"2024-04-12T22:36:47.987359644Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (cluster:master_nodes and on (node) cluster:infra_nodes)\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000132182,\"lastEvaluation\":\"2024-04-12T22:36:47.964024441Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"label_node_role_kubernetes_io\":\"master\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000325696,\"lastEvaluation\":\"2024-04-12T22:36:47.963525342Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage:ratio\",\"query\":\"sum by (cluster) (container_memory_usage_bytes{container=\\\"\\\",pod!=\\\"\\\"}) / sum by (cluster) (machine_memory_bytes)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010134309,\"lastEvaluation\":\"2024-04-12T22:36:47.933709305Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage_bytes:sum\",\"query\":\"sum(node_memory_MemTotal_bytes{job=\\\"node-exporter\\\"} - node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000199454,\"lastEvaluation\":\"2024-04-12T22:36:47.967467268Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_instance_type_count:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) (cluster:master_nodes or on (node) kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000256065,\"lastEvaluation\":\"2024-04-12T22:36:47.985698522Z\",\"type\":\"recording\"},{\"name\":\"cluster:nodes_roles\",\"query\":\"cluster:master_infra_nodes or on (node) cluster:master_nodes or on (node) cluster:infra_nodes or on (node) max without (endpoint, instance, job, pod, service) (kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000198834,\"lastEvaluation\":\"2024-04-12T22:36:47.964157634Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:containers:sum\",\"query\":\"sum(max by (instance) (kubelet_containers_per_pod_count_sum))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000128573,\"lastEvaluation\":\"2024-04-12T22:36:47.994546103Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_node_ready:avg5m\",\"query\":\"avg_over_time((count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1)) / scalar(count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003232883,\"lastEvaluation\":\"2024-04-12T22:36:48.004967685Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_schedulable_node_ready_reachable:avg5m\",\"query\":\"avg_over_time((((count((max by (node) (up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1) and max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1) and min by (node) (kube_node_spec_unschedulable == 0))) / scalar(count(min by (node) (kube_node_spec_unschedulable == 0))))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006318392,\"lastEvaluation\":\"2024-04-12T22:36:47.998647403Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace=~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001252884,\"lastEvaluation\":\"2024-04-12T22:36:48.022030176Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:pods:terminal:workload:sum\",\"query\":\"count(count by (namespace, pod) (kube_pod_restart_policy{namespace!~\\\"openshift-.+\\\",type!=\\\"Always\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000127912,\"lastEvaluation\":\"2024-04-12T22:36:47.994417121Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:resources:sum\",\"query\":\"topk(500, max by (resource) (apiserver_storage_objects != -1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002758324,\"lastEvaluation\":\"2024-04-12T22:36:47.991656857Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:max:5m\",\"query\":\"max_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000062601,\"lastEvaluation\":\"2024-04-12T22:36:47.987296113Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:min:5m\",\"query\":\"min_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000095062,\"lastEvaluation\":\"2024-04-12T22:36:47.987200051Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace!~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001740804,\"lastEvaluation\":\"2024-04-12T22:36:48.02328452Z\",\"type\":\"recording\"},{\"name\":\"cluster:virt_platform_nodes:sum\",\"query\":\"count by (type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (sum by (instance, type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (virt_platform))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000180903,\"lastEvaluation\":\"2024-04-12T22:36:47.964707865Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_object_counts:sum\",\"query\":\"sum by (instance) (apiserver_storage_objects != -1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003892765,\"lastEvaluation\":\"2024-04-12T22:36:47.987761702Z\",\"type\":\"recording\"},{\"name\":\"kube_running_pod_ready\",\"query\":\"(max without (condition, container, endpoint, instance, job, service) (((kube_pod_status_ready{condition=\\\"false\\\"} == 1) * 0 or (kube_pod_status_ready{condition=\\\"true\\\"} == 1)) * on (pod, namespace) group_left () group by (pod, namespace) (kube_pod_status_phase{phase=~\\\"Running|Unknown|Pending\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013824628,\"lastEvaluation\":\"2024-04-12T22:36:48.008202298Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_cpu_usage:sum\",\"query\":\"sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=\\\"\\\",container!=\\\"POD\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007228534,\"lastEvaluation\":\"2024-04-12T22:36:47.92647816Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_memory_usage_bytes:sum\",\"query\":\"sum by (namespace) (container_memory_usage_bytes{container!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009425742,\"lastEvaluation\":\"2024-04-12T22:36:47.917049358Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_cores:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000163023,\"lastEvaluation\":\"2024-04-12T22:36:47.994675856Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_sockets:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (max by (node, package, label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000167663,\"lastEvaluation\":\"2024-04-12T22:36:47.994985912Z\",\"type\":\"recording\"},{\"name\":\"openshift:cpu_usage_cores:sum\",\"query\":\"cluster:cpu_usage_cores:sum - workload:cpu_usage_cores:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000148032,\"lastEvaluation\":\"2024-04-12T22:36:47.975989784Z\",\"type\":\"recording\"},{\"name\":\"openshift:memory_usage_bytes:sum\",\"query\":\"cluster:memory_usage_bytes:sum - workload:memory_usage_bytes:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000188714,\"lastEvaluation\":\"2024-04-12T22:36:47.985507768Z\",\"type\":\"recording\"},{\"name\":\"pod:container_cpu_usage:sum\",\"query\":\"sum by (pod, namespace) (rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010408544,\"lastEvaluation\":\"2024-04-12T22:36:47.895636169Z\",\"type\":\"recording\"},{\"name\":\"pod:container_fs_usage_bytes:sum\",\"query\":\"sum by (pod, namespace) (container_fs_usage_bytes{pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010997915,\"lastEvaluation\":\"2024-04-12T22:36:47.906047623Z\",\"type\":\"recording\"},{\"name\":\"profile:cluster_monitoring_operator_collection_profile:max\",\"query\":\"max by (profile) (cluster_monitoring_operator_collection_profile == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000066711,\"lastEvaluation\":\"2024-04-12T22:36:47.996011892Z\",\"type\":\"recording\"},{\"name\":\"workload:capacity_physical_cpu_cores:sum\",\"query\":\"(sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_infra=\\\"\\\",label_node_role_kubernetes_io_master=\\\"\\\"} or absent(__does_not_exist__) * 0)) + ((sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_master=\\\"true\\\"} or absent(__does_not_exist__) * 0) * ((max(cluster_master_schedulable == 1) * 0 + 1) or (absent(cluster_master_schedulable == 1) * 0))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00048553,\"lastEvaluation\":\"2024-04-12T22:36:47.986712971Z\",\"type\":\"recording\"},{\"name\":\"workload:cpu_usage_cores:sum\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008318751,\"lastEvaluation\":\"2024-04-12T22:36:47.967667942Z\",\"type\":\"recording\"},{\"name\":\"workload:memory_usage_bytes:sum\",\"query\":\"sum(container_memory_working_set_bytes{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009365742,\"lastEvaluation\":\"2024-04-12T22:36:47.976139526Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.130430439,\"lastEvaluation\":\"2024-04-12T22:36:47.895141616Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-monitoring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"monitoring:container_memory_working_set_bytes:sum\",\"query\":\"sum by (namespace) (max without (instance) (container_memory_working_set_bytes{container=\\\"\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000993363,\"lastEvaluation\":\"2024-04-12T22:36:30.768302406Z\",\"type\":\"recording\"},{\"name\":\"monitoring:haproxy_server_http_responses_total:sum\",\"query\":\"sum by (exported_service) (rate(haproxy_server_http_responses_total{exported_namespace=\\\"openshift-monitoring\\\",exported_service=~\\\"alertmanager-main|prometheus-k8s\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000488686,\"lastEvaluation\":\"2024-04-12T22:36:30.772285858Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_samples_post_metric_relabeling:topk3\",\"query\":\"topk(3, max by (namespace, job) (topk by (namespace, job) (1, scrape_samples_post_metric_relabeling)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00095385,\"lastEvaluation\":\"2024-04-12T22:36:30.771330297Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_series_added:topk3_sum1h\",\"query\":\"topk(3, sum by (namespace, job) (sum_over_time(scrape_series_added[1h])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002031528,\"lastEvaluation\":\"2024-04-12T22:36:30.769297129Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicationController\\\"}, \\\"replicationcontroller\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicationcontroller, namespace) group_left (owner_name) topk by (replicationcontroller, namespace) (1, max by (replicationcontroller, namespace, owner_name) (kube_replicationcontroller_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deploymentconfig\"},\"health\":\"ok\",\"evaluationTime\":0.000470985,\"lastEvaluation\":\"2024-04-12T22:36:30.772775774Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_samples_appended_total:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (rate(prometheus_tsdb_head_samples_appended_total{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[2m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000264694,\"lastEvaluation\":\"2024-04-12T22:36:30.768035902Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_series:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (prometheus_tsdb_head_series{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000344929,\"lastEvaluation\":\"2024-04-12T22:36:30.767688283Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.005674618,\"lastEvaluation\":\"2024-04-12T22:36:30.767033763Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-sre.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"code:apiserver_request_total:rate:sum\",\"query\":\"sum by (code) (rate(apiserver_request_total{job=\\\"apiserver\\\"}[10m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.019782155,\"lastEvaluation\":\"2024-04-12T22:36:31.411271461Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.022887027,\"lastEvaluation\":\"2024-04-12T22:36:31.410800849Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-vsphere.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:vsphere_csi_migration:max\",\"query\":\"max by (status) (vsphere_csi_migration{status=~\\\"|LegacyDeprecatedInTreeDriver|CSIWithMigrationDriver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000243123,\"lastEvaluation\":\"2024-04-12T22:36:26.80731085Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_esxi_version_total:sum\",\"query\":\"sum by (version) (vsphere_esxi_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000080114,\"lastEvaluation\":\"2024-04-12T22:36:26.807030775Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_infrastructure_failure_domains:max\",\"query\":\"max by (scope) (vsphere_infrastructure_failure_domains)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000066604,\"lastEvaluation\":\"2024-04-12T22:36:26.807243016Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_node_hw_version_total:sum\",\"query\":\"sum by (hw_version) (vsphere_node_hw_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000065644,\"lastEvaluation\":\"2024-04-12T22:36:26.807112469Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_topology_tags:max\",\"query\":\"max by (source) (vsphere_topology_tags)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000062223,\"lastEvaluation\":\"2024-04-12T22:36:26.807179523Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_vcenter_info:sum\",\"query\":\"sum by (version, build) (vsphere_vcenter_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000231123,\"lastEvaluation\":\"2024-04-12T22:36:26.806797263Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000769051,\"lastEvaluation\":\"2024-04-12T22:36:26.806787552Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-state-metrics\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kube-state-metrics-rules-07cb1c2d-024f-4864-ad8b-21ee1306da90.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeStateMetricsListErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in list operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in list operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000525768,\"lastEvaluation\":\"2024-04-12T22:36:24.944480735Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStateMetricsWatchErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in watch operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in watch operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000445704,\"lastEvaluation\":\"2024-04-12T22:36:24.945008353Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001001923,\"lastEvaluation\":\"2024-04-12T22:36:24.944454774Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_cpu_usage_seconds_total\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate\",\"query\":\"sum by (cluster, namespace, pod, container) (irate(container_cpu_usage_seconds_total{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"}[5m])) * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012626512,\"lastEvaluation\":\"2024-04-12T22:36:47.746380067Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.012637473,\"lastEvaluation\":\"2024-04-12T22:36:47.746372897Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_cache\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_cache\",\"query\":\"container_memory_cache{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010943622,\"lastEvaluation\":\"2024-04-12T22:36:35.623483087Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.010261186,\"lastEvaluation\":\"2024-04-12T22:36:35.622071511Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_rss\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_rss\",\"query\":\"container_memory_rss{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013819512,\"lastEvaluation\":\"2024-04-12T22:36:22.912383576Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.01146525,\"lastEvaluation\":\"2024-04-12T22:36:22.911027189Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_swap\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_swap\",\"query\":\"container_memory_swap{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011071578,\"lastEvaluation\":\"2024-04-12T22:36:38.910817703Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.010704188,\"lastEvaluation\":\"2024-04-12T22:36:38.910588856Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_working_set_bytes\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_working_set_bytes\",\"query\":\"container_memory_working_set_bytes{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012711706,\"lastEvaluation\":\"2024-04-12T22:36:50.154884787Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.012724857,\"lastEvaluation\":\"2024-04-12T22:36:50.154876566Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_resource\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004433776,\"lastEvaluation\":\"2024-04-12T22:36:42.149576059Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006873023,\"lastEvaluation\":\"2024-04-12T22:36:42.126663768Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004633216,\"lastEvaluation\":\"2024-04-12T22:36:42.140182459Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009029006,\"lastEvaluation\":\"2024-04-12T22:36:42.109512855Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005386616,\"lastEvaluation\":\"2024-04-12T22:36:42.154011685Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006560217,\"lastEvaluation\":\"2024-04-12T22:36:42.133539481Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004755714,\"lastEvaluation\":\"2024-04-12T22:36:42.144817696Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008116488,\"lastEvaluation\":\"2024-04-12T22:36:42.11854467Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.047212386,\"lastEvaluation\":\"2024-04-12T22:36:42.109503685Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.pod_owner\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"DaemonSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"daemonset\"},\"health\":\"ok\",\"evaluationTime\":0.001090018,\"lastEvaluation\":\"2024-04-12T22:36:38.838936232Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicaSet\\\"}, \\\"replicaset\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicaset, namespace) group_left (owner_name) topk by (replicaset, namespace) (1, max by (replicaset, namespace, owner_name) (kube_replicaset_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deployment\"},\"health\":\"ok\",\"evaluationTime\":0.002620491,\"lastEvaluation\":\"2024-04-12T22:36:38.835415538Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"Job\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"job\"},\"health\":\"ok\",\"evaluationTime\":0.000240513,\"lastEvaluation\":\"2024-04-12T22:36:38.840454482Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"StatefulSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"statefulset\"},\"health\":\"ok\",\"evaluationTime\":0.000424552,\"lastEvaluation\":\"2024-04-12T22:36:38.84002826Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.005396507,\"lastEvaluation\":\"2024-04-12T22:36:38.835301078Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-scheduler.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000126387,\"lastEvaluation\":\"2024-04-12T22:36:42.25654512Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000113555,\"lastEvaluation\":\"2024-04-12T22:36:42.255835723Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000120876,\"lastEvaluation\":\"2024-04-12T22:36:42.255011989Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000096905,\"lastEvaluation\":\"2024-04-12T22:36:42.255950319Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000138198,\"lastEvaluation\":\"2024-04-12T22:36:42.255134295Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000271785,\"lastEvaluation\":\"2024-04-12T22:36:42.254155403Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000494666,\"lastEvaluation\":\"2024-04-12T22:36:42.256048034Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000559229,\"lastEvaluation\":\"2024-04-12T22:36:42.255274063Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000580281,\"lastEvaluation\":\"2024-04-12T22:36:42.254429078Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002529355,\"lastEvaluation\":\"2024-04-12T22:36:42.254145093Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubelet.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000744184,\"lastEvaluation\":\"2024-04-12T22:36:46.115656106Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000682296,\"lastEvaluation\":\"2024-04-12T22:36:46.114953035Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001092848,\"lastEvaluation\":\"2024-04-12T22:36:46.113858227Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002420079,\"lastEvaluation\":\"2024-04-12T22:36:46.113845676Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-apps\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"KubeContainerWaiting\",\"query\":\"sum by (namespace, pod, container, cluster) (kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"pod/{{ $labels.pod }} in namespace {{ $labels.namespace }} on container {{ $labels.container}} has been in waiting state for longer than 1 hour.\",\"summary\":\"Pod container waiting longer than 1 hour\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubeContainerWaiting\",\"container\":\"node-probe\",\"namespace\":\"openshift-must-gather-dxffw\",\"pod\":\"perf-node-gather-daemonset-b5xc5\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"pod/perf-node-gather-daemonset-b5xc5 in namespace openshift-must-gather-dxffw on container node-probe has been in waiting state for longer than 1 hour.\",\"summary\":\"Pod container waiting longer than 1 hour\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000662295,\"lastEvaluation\":\"2024-04-12T22:36:28.236250899Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetMisScheduled\",\"query\":\"kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.\",\"summary\":\"DaemonSet pods are misscheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000344848,\"lastEvaluation\":\"2024-04-12T22:36:28.237572849Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetNotScheduled\",\"query\":\"kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} - kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.\",\"summary\":\"DaemonSet pods are not scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000657115,\"lastEvaluation\":\"2024-04-12T22:36:28.236914854Z\",\"type\":\"alerting\"},{\"state\":\"pending\",\"name\":\"KubeDaemonSetRolloutStuck\",\"query\":\"((kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != 0) or (kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} has not finished or progressed for at least 30 minutes.\",\"summary\":\"DaemonSet rollout is stuck.\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubeDaemonSetRolloutStuck\",\"container\":\"kube-rbac-proxy-main\",\"daemonset\":\"perf-node-gather-daemonset\",\"endpoint\":\"https-main\",\"job\":\"kube-state-metrics\",\"namespace\":\"openshift-must-gather-dxffw\",\"service\":\"kube-state-metrics\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"DaemonSet openshift-must-gather-dxffw/perf-node-gather-daemonset has not finished or progressed for at least 30 minutes.\",\"summary\":\"DaemonSet rollout is stuck.\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.002392867,\"lastEvaluation\":\"2024-04-12T22:36:28.233856691Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentGenerationMismatch\",\"query\":\"kube_deployment_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_deployment_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.\",\"summary\":\"Deployment generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001193103,\"lastEvaluation\":\"2024-04-12T22:36:28.228656795Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentRolloutStuck\",\"query\":\"kube_deployment_status_condition{condition=\\\"Progressing\\\",job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",status=\\\"false\\\"} != 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Rollout of deployment {{ $labels.namespace }}/{{ $labels.deployment }} is not progressing for longer than 15 minutes.\",\"summary\":\"Deployment rollout is not progressing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000670776,\"lastEvaluation\":\"2024-04-12T22:36:28.229851058Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaMaxedOut\",\"query\":\"kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has been running at max replicas for longer than 15 minutes.\",\"summary\":\"HPA is running at max replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000459619,\"lastEvaluation\":\"2024-04-12T22:36:28.24056117Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaReplicasMismatch\",\"query\":\"(kube_horizontalpodautoscaler_status_desired_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_horizontalpodautoscaler_spec_min_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003c kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and changes(kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[15m]) == 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has not matched the desired number of replicas for longer than 15 minutes.\",\"summary\":\"HPA has not matched desired number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001479378,\"lastEvaluation\":\"2024-04-12T22:36:28.239080822Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobFailed\",\"query\":\"kube_job_failed{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeJobFailed.md\",\"summary\":\"Job failed to complete.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000314316,\"lastEvaluation\":\"2024-04-12T22:36:28.238765466Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobNotCompleted\",\"query\":\"time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} and kube_job_status_active{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0) \\u003e 43200\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than {{ \\\"43200\\\" | humanizeDuration }} to complete.\",\"summary\":\"Job did not complete in time\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000697987,\"lastEvaluation\":\"2024-04-12T22:36:28.237918717Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodCrashLooping\",\"query\":\"max_over_time(kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",reason=\\\"CrashLoopBackOff\\\"}[5m]) \\u003e= 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is in waiting state (reason: \\\"CrashLoopBackOff\\\").\",\"summary\":\"Pod is crash looping.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000842105,\"lastEvaluation\":\"2024-04-12T22:36:28.220530281Z\",\"type\":\"alerting\"},{\"state\":\"pending\",\"name\":\"KubePodNotReady\",\"query\":\"sum by (namespace, pod, cluster) (max by (namespace, pod, cluster) (kube_pod_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Pending|Unknown\\\"} unless ignoring (phase) (kube_pod_status_unschedulable{job=\\\"kube-state-metrics\\\"} == 1)) * on (namespace, pod, cluster) group_left (owner_kind) topk by (namespace, pod, cluster) (1, max by (namespace, pod, owner_kind, cluster) (kube_pod_owner{owner_kind!=\\\"Job\\\"}))) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePodNotReady.md\",\"summary\":\"Pod has been in a non-ready state for more than 15 minutes.\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubePodNotReady\",\"namespace\":\"openshift-must-gather-dxffw\",\"pod\":\"perf-node-gather-daemonset-b5xc5\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod openshift-must-gather-dxffw/perf-node-gather-daemonset-b5xc5 has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePodNotReady.md\",\"summary\":\"Pod has been in a non-ready state for more than 15 minutes.\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.007222574,\"lastEvaluation\":\"2024-04-12T22:36:28.221432431Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetGenerationMismatch\",\"query\":\"kube_statefulset_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.\",\"summary\":\"StatefulSet generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000705964,\"lastEvaluation\":\"2024-04-12T22:36:28.231651031Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetReplicasMismatch\",\"query\":\"(kube_statefulset_status_replicas_ready{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[10m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.\",\"summary\":\"StatefulSet has not matched the expected number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001109779,\"lastEvaluation\":\"2024-04-12T22:36:28.230522894Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetUpdateNotRolledOut\",\"query\":\"(max without (revision) (kube_statefulset_status_current_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} unless kube_statefulset_status_update_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) * (kube_statefulset_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.\",\"summary\":\"StatefulSet update has not been rolled out.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00162914,\"lastEvaluation\":\"2024-04-12T22:36:28.232358075Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.020364443,\"lastEvaluation\":\"2024-04-12T22:36:28.220445469Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-resources\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeCPUOvercommit\",\"query\":\"sum by (cluster) (namespace_cpu:kube_pod_container_resource_requests:sum{job=\\\"kube-state-metrics\\\"}) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted CPU resource requests for Pods by {{ $value }} CPU shares and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted CPU resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558299,\"lastEvaluation\":\"2024-04-12T22:36:38.436962344Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeMemoryOvercommit\",\"query\":\"sum by (cluster) (namespace_memory:kube_pod_container_resource_requests:sum) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted memory resource requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted memory resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558559,\"lastEvaluation\":\"2024-04-12T22:36:38.437522304Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaAlmostFull\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) \\u003e 0.9 \\u003c 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota is going to be full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001061567,\"lastEvaluation\":\"2024-04-12T22:36:38.438082153Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaExceeded\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) \\u003e 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota has exceeded the limits.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000732379,\"lastEvaluation\":\"2024-04-12T22:36:38.439862808Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaFullyUsed\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota is fully used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000716728,\"lastEvaluation\":\"2024-04-12T22:36:38.43914473Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003643934,\"lastEvaluation\":\"2024-04-12T22:36:38.436952983Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-storage\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeErrors\",\"query\":\"kube_persistentvolume_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Failed|Pending\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The persistent volume {{ $labels.persistentvolume }} has status {{ $labels.phase }}.\",\"summary\":\"PersistentVolume is having issues with provisioning.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000957141,\"lastEvaluation\":\"2024-04-12T22:36:33.212259431Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeFillingUp\",\"query\":\"(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_capacity_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.03 and kubelet_volume_stats_used_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage }} free.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeFillingUp.md\",\"summary\":\"PersistentVolume is filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001723601,\"lastEvaluation\":\"2024-04-12T22:36:33.206464493Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeFillingUp\",\"query\":\"(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_capacity_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.15 and kubelet_volume_stats_used_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 and predict_linear(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[6h], 4 * 24 * 3600) \\u003c 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) k{\"status\":\"success\",\"data\":{\"groups\":[{\"name\":\"CloudCredentialOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-59f29d20-14e3-45ac-95b2-817cbe434ddc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorDeprovisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsDeprovisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be cleaned up\",\"summary\":\"One or more CredentialsRequest CRs are unable to be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000117926,\"lastEvaluation\":\"2024-04-12T22:36:31.735304389Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorInsufficientCloudCreds\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"InsufficientCloudCreds\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.\",\"message\":\"Cluster's cloud credentials insufficient for minting or passthrough\",\"summary\":\"Problem with the available platform credentials.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000108965,\"lastEvaluation\":\"2024-04-12T22:36:31.735423516Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorProvisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsProvisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be fulfilled\",\"summary\":\"One or more CredentialsRequest CRs are unable to be processed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000098825,\"lastEvaluation\":\"2024-04-12T22:36:31.735204354Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorStaleCredentials\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"StaleCredentials\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.\",\"message\":\"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).\",\"summary\":\"One or more CredentialsRequest CRs are stale and should be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000105386,\"lastEvaluation\":\"2024-04-12T22:36:31.735533862Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorTargetNamespaceMissing\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"MissingTargetNamespace\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.\",\"message\":\"CredentialsRequest(s) pointing to non-existent namespace\",\"summary\":\"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000150643,\"lastEvaluation\":\"2024-04-12T22:36:31.735039966Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000623703,\"lastEvaluation\":\"2024-04-12T22:36:31.735017464Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-machine-approver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-02d4c4c8-9481-4482-a4eb-aa2196fb1abd.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineApproverMaxPendingCSRsReached\",\"query\":\"mapi_current_pending_csr \\u003e mapi_max_pending_csr\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of pending CertificateSigningRequests has exceeded the\\nmaximum threshold (current number of machine + 100). Check the\\npending CSRs to determine which machines need approval, also check\\nthat the nodelink controller is running in the openshift-machine-api\\nnamespace.\\n\",\"summary\":\"max pending CSRs threshold reached.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000264985,\"lastEvaluation\":\"2024-04-12T22:36:35.040054675Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000289416,\"lastEvaluation\":\"2024-04-12T22:36:35.040034635Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-tuning-operator.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-475016e5-fc0a-4af1-bd38-a45d26c2b85a.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NTODegraded\",\"query\":\"nto_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Node Tuning Operator is degraded. Review the \\\"node-tuning\\\" ClusterOperator object for further details.\",\"summary\":\"The Node Tuning Operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000087741,\"lastEvaluation\":\"2024-04-12T22:36:35.756300122Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NTOPodsNotReady\",\"query\":\"kube_pod_status_ready{condition=\\\"true\\\",namespace=\\\"openshift-cluster-node-tuning-operator\\\"} == 0\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.pod }} is not ready.\\nReview the \\\"Event\\\" objects in \\\"openshift-cluster-node-tuning-operator\\\" namespace for further details.\\n\",\"summary\":\"Pod {{ $labels.pod }} is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000291586,\"lastEvaluation\":\"2024-04-12T22:36:35.756006506Z\",\"type\":\"alerting\"},{\"name\":\"nto_custom_profiles:count\",\"query\":\"count by (_id) (nto_profile_calculated_total{profile!~\\\"openshift\\\",profile!~\\\"openshift-control-plane\\\",profile!~\\\"openshift-node\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000113402,\"lastEvaluation\":\"2024-04-12T22:36:35.756391654Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.00050883,\"lastEvaluation\":\"2024-04-12T22:36:35.755999396Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"SamplesOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-5c7cc52f-a9cb-47fe-a845-20e9f9a26649.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SamplesDegraded\",\"query\":\"openshift_samples_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples could not be deployed and the operator is degraded. Review the \\\"openshift-samples\\\" ClusterOperator object for further details.\\n\",\"summary\":\"Samples operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000093025,\"lastEvaluation\":\"2024-04-12T22:36:47.104513835Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesImagestreamImportFailing\",\"query\":\"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \\u003e sum(openshift_samples_failed_imagestream_import_info)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports.  You can look at the \\\"openshift-samples\\\"\\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is detecting problems with imagestream image imports\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000264614,\"lastEvaluation\":\"2024-04-12T22:36:47.10424756Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesInvalidConfig\",\"query\":\"openshift_samples_invalidconfig_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator has been given an invalid configuration.\\n\",\"summary\":\"Samples operator Invalid configuration\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077034,\"lastEvaluation\":\"2024-04-12T22:36:47.10460794Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingSecret\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_secret\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator cannot find the samples pull secret in the openshift namespace.\\n\",\"summary\":\"Samples operator is not able to find secret\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000078804,\"lastEvaluation\":\"2024-04-12T22:36:47.104685694Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingTBRCredential\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_tbr_credential\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\\n\",\"summary\":\"Samples operator is not able to find the credentials for registry\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000072294,\"lastEvaluation\":\"2024-04-12T22:36:47.104765228Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesRetriesMissingOnImagestreamImportFailing\",\"query\":\"sum(openshift_samples_failed_imagestream_import_info) \\u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\\nimports are not occurring.  Contact support.  You can look at the \\\"openshift-samples\\\" ClusterOperator object\\nfor details. Most likely there are issues with the external image registry hosting the images that need to\\nbe investigated.  The list of ImageStreams that have failing imports are:\\n{{ range query \\\"openshift_samples_failed_imagestream_import_info \\u003e 0\\\" }}\\n  {{ .Labels.name }}\\n{{ end }}\\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is having problems with imagestream imports and its retries.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000430863,\"lastEvaluation\":\"2024-04-12T22:36:47.103814857Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesTBRInaccessibleOnBoot\",\"query\":\"openshift_samples_tbr_inaccessible_info == 1\",\"duration\":172800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"One of two situations has occurred.  Either\\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\\nassociated with each image stream tag from the samples catalog is\\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\\nassist the mirroring process.\\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\\n\",\"summary\":\"Samples operator is not able to access the registry on boot\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000079234,\"lastEvaluation\":\"2024-04-12T22:36:47.104838392Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001210843,\"lastEvaluation\":\"2024-04-12T22:36:47.103443407Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"default-storage-classes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MultipleDefaultStorageClasses\",\"query\":\"max_over_time(default_storage_class_count[5m]) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster storage operator monitors all storage classes configured in the cluster\\nand checks there is not more than one default StorageClass configured.\\n\",\"message\":\"StorageClass count check is failing (there should not be more than one default StorageClass)\",\"summary\":\"More than one default StorageClass detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000251395,\"lastEvaluation\":\"2024-04-12T22:36:37.477974261Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000260775,\"lastEvaluation\":\"2024-04-12T22:36:37.477967861Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodStartupStorageOperationsFailing\",\"query\":\"increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status!=\\\"success\\\"}[5m]) \\u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status=\\\"success\\\"}[5m]) == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Failing storage operation \\\"{{ $labels.operation_name }}\\\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\\nfrom starting for past 5 minutes.\\nPlease investigate Pods that are \\\"ContainerCreating\\\" on the node: \\\"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\\\".\\nEvents of the Pods should contain exact error message: \\\"oc describe pod -n \\u003cpod namespace\\u003e \\u003cpod name\\u003e\\\".\\n\",\"summary\":\"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003662945,\"lastEvaluation\":\"2024-04-12T22:36:49.473150763Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001723513,\"lastEvaluation\":\"2024-04-12T22:36:49.454522558Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-selinux.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"name\":\"cluster:volume_manager_selinux_pod_context_mismatch_total\",\"query\":\"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000381297,\"lastEvaluation\":\"2024-04-12T22:36:28.076271377Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volume_context_mismatch_errors_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000109212,\"lastEvaluation\":\"2024-04-12T22:36:28.076799817Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000142513,\"lastEvaluation\":\"2024-04-12T22:36:28.076655824Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volumes_admitted_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000286085,\"lastEvaluation\":\"2024-04-12T22:36:28.076910049Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000935567,\"lastEvaluation\":\"2024-04-12T22:36:28.076262836Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-operators\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-b7ec8a0e-ca37-48ff-91bd-48b44cd606d6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotEvaluateConditionalUpdates\",\"query\":\"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\\\"Recommended\\\",status=\\\"Unknown\\\"}) \\u003e= 3600)\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.\",\"summary\":\"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000149303,\"lastEvaluation\":\"2024-04-12T22:36:25.502645639Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterNotUpgradeable\",\"query\":\"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\\\"Upgradeable\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"Upgradeable\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0) (ne (len (label \\\"reason\\\" $value)) 0) }}{{label \\\"reason\\\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000282415,\"lastEvaluation\":\"2024-04-12T22:36:25.50115166Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDegraded\",\"query\":\"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\\\"Degraded\\\",job=\\\"cluster-version-operator\\\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\\\"cluster-version-operator\\\"})) == 1)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has been degraded for 30 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000644752,\"lastEvaluation\":\"2024-04-12T22:36:25.501743902Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDown\",\"query\":\"max by (namespace, name, reason) (cluster_operator_up{job=\\\"cluster-version-operator\\\"} == 0)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has not been available for 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000306316,\"lastEvaluation\":\"2024-04-12T22:36:25.501435936Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorFlapping\",\"query\":\"max by (namespace, name) (changes(cluster_operator_up{job=\\\"cluster-version-operator\\\"}[2m]) \\u003e 2)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator up status is changing often.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000254825,\"lastEvaluation\":\"2024-04-12T22:36:25.502389944Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001803656,\"lastEvaluation\":\"2024-04-12T22:36:25.500984548Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-b7ec8a0e-ca37-48ff-91bd-48b44cd606d6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotRetrieveUpdates\",\"query\":\"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \\u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\\\"RetrievedUpdates\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\",reason!=\\\"NoChannel\\\"}))\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"RetrievedUpdates\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0)}}{{label \\\"reason\\\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\\\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\\\n'}\\\"`{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000241142,\"lastEvaluation\":\"2024-04-12T22:36:39.739623191Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterReleaseNotAccepted\",\"query\":\"max by (namespace, name, reason) (cluster_operator_conditions{condition=\\\"ReleaseAccepted\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"The desired cluster release has not been accepted for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000125806,\"lastEvaluation\":\"2024-04-12T22:36:39.73996714Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterVersionOperatorDown\",\"query\":\"absent(up{job=\\\"cluster-version-operator\\\"} == 1)\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-cluster-version\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} For more information refer to {{ label \\\"url\\\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}\",\"summary\":\"Cluster version operator has disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000237812,\"lastEvaluation\":\"2024-04-12T22:36:39.739383739Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UpdateAvailable\",\"query\":\"sum by (channel, namespace, upstream) (cluster_version_available_updates) \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Your upstream update recommendation service recommends you update your cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000098335,\"lastEvaluation\":\"2024-04-12T22:36:39.739867784Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000719859,\"lastEvaluation\":\"2024-04-12T22:36:39.739375598Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift/console-operator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-13a53b5a-afd9-4211-9680-fe10e32351a7.yaml\",\"rules\":[{\"name\":\"cluster:console_auth_login_failures_total:sum\",\"query\":\"sum by (reason) (console_auth_login_failures_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000116322,\"lastEvaluation\":\"2024-04-12T22:36:29.799875967Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_requests_total:sum\",\"query\":\"sum(console_auth_login_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000191814,\"lastEvaluation\":\"2024-04-12T22:36:29.799544241Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_successes_total:sum\",\"query\":\"sum by (role) (console_auth_login_successes_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000133083,\"lastEvaluation\":\"2024-04-12T22:36:29.799738254Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_logout_requests_total:sum\",\"query\":\"sum by (reason) (console_auth_logout_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000080572,\"lastEvaluation\":\"2024-04-12T22:36:29.799993909Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_customization_perspectives_info:max\",\"query\":\"max by (name, state) (console_customization_perspectives_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000078672,\"lastEvaluation\":\"2024-04-12T22:36:29.800314685Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_plugins_info:max\",\"query\":\"max by (name, state) (console_plugins_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000163453,\"lastEvaluation\":\"2024-04-12T22:36:29.800149772Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_usage_users:max\",\"query\":\"max by (role) (console_usage_users)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000072861,\"lastEvaluation\":\"2024-04-12T22:36:29.800076061Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000905939,\"lastEvaluation\":\"2024-04-12T22:36:29.798943678Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-dns.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-be119138-53ca-458e-a33e-816e88c6ca38.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CoreDNSErrorsHigh\",\"query\":\"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\\\"SERVFAIL\\\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.\",\"summary\":\"CoreDNS serverfail\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000269855,\"lastEvaluation\":\"2024-04-12T22:36:51.98821851Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSHealthCheckSlow\",\"query\":\"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \\u003e 10\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})\",\"summary\":\"CoreDNS health checks\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000361857,\"lastEvaluation\":\"2024-04-12T22:36:51.987855743Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSPanicking\",\"query\":\"increase(coredns_panics_total[10m]) \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}\",\"summary\":\"CoreDNS panic\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000244395,\"lastEvaluation\":\"2024-04-12T22:36:51.987609808Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001073007,\"lastEvaluation\":\"2024-04-12T22:36:51.983556814Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"etcd\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-bdafb11d-bf7f-4480-ae4e-7cbeddeea773.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdDatabaseHighFragmentationRatio\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \\u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \\u003e 1.048576e+08\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.\",\"runbook_url\":\"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation\",\"summary\":\"etcd database size in use is less than 50% of the actual allocated storage.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000273924,\"lastEvaluation\":\"2024-04-12T22:36:44.199758574Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdDatabaseQuotaLowSpace\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \\u003e 95\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md\",\"summary\":\"etcd cluster database is running full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000176359,\"lastEvaluation\":\"2024-04-12T22:36:44.199396135Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdExcessiveDatabaseGrowth\",\"query\":\"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \\u003e etcd_server_quota_backend_bytes\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.\",\"summary\":\"etcd cluster database growing very fast.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.0001838,\"lastEvaluation\":\"2024-04-12T22:36:44.199573524Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighCommitDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.25\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile commit durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00036941,\"lastEvaluation\":\"2024-04-12T22:36:44.199025365Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000312917,\"lastEvaluation\":\"2024-04-12T22:36:44.198711528Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.5\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00038715,\"lastEvaluation\":\"2024-04-12T22:36:44.198323418Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedProposals\",\"query\":\"rate(etcd_server_proposals_failed_total{job=~\\\".*etcd.*\\\"}[15m]) \\u003e 5\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of proposal failures.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000140508,\"lastEvaluation\":\"2024-04-12T22:36:44.19818184Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMemberCommunicationSlow\",\"query\":\"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.15\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster member communication is slow.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000718188,\"lastEvaluation\":\"2024-04-12T22:36:44.197462382Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMembersDown\",\"query\":\"max without (endpoint) (sum without (instance) (up{job=~\\\".*etcd.*\\\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\\\".*etcd.*\\\"}[2m])) \\u003e 0.01)) \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": members are down ({{ $value }}).\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md\",\"summary\":\"etcd cluster members are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000461165,\"lastEvaluation\":\"2024-04-12T22:36:44.196853459Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdNoLeader\",\"query\":\"etcd_server_has_leader{job=~\\\".*etcd.*\\\"} == 0\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member {{ $labels.instance }} has no leader.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md\",\"summary\":\"etcd cluster has no leader.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000144798,\"lastEvaluation\":\"2024-04-12T22:36:44.197316204Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003145091,\"lastEvaluation\":\"2024-04-12T22:36:44.196670111Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-bdafb11d-bf7f-4480-ae4e-7cbeddeea773.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdGRPCRequestsSlow\",\"query\":\"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\\\"Defragment\\\",grpc_type=\\\"unary\\\",job=\\\"etcd\\\"}[10m]))) \\u003e 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md\",\"summary\":\"etcd grpc requests are slow\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.008333211,\"lastEvaluation\":\"2024-04-12T22:36:54.417336925Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 50\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.013154615,\"lastEvaluation\":\"2024-04-12T22:36:54.441269989Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 10\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.014937774,\"lastEvaluation\":\"2024-04-12T22:36:54.425723935Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfLeaderChanges\",\"query\":\"avg(changes(etcd_server_is_leader[10m])) \\u003e 5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.\",\"summary\":\"etcd cluster has high number of leader changes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000160483,\"lastEvaluation\":\"2024-04-12T22:36:54.454426404Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdInsufficientMembers\",\"query\":\"sum without (instance, pod) (up{job=\\\"etcd\\\"} == bool 1 and etcd_server_has_leader{job=\\\"etcd\\\"} == bool 1) \\u003c ((count without (instance, pod) (up{job=\\\"etcd\\\"}) + 1) / 2)\",\"duration\":180,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md\",\"summary\":\"etcd is reporting that a majority of instances are unavailable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000322627,\"lastEvaluation\":\"2024-04-12T22:36:54.454587997Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.037585688,\"lastEvaluation\":\"2024-04-12T22:36:54.417327345Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imageregistry.operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-2e046cbc-5e3f-419b-9e59-085c6eb3d543.yaml\",\"rules\":[{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.Create\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000205973,\"lastEvaluation\":\"2024-04-12T22:36:27.142541754Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.ServeBlob\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000348556,\"lastEvaluation\":\"2024-04-12T22:36:27.142191437Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Get\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000202683,\"lastEvaluation\":\"2024-04-12T22:36:27.142749038Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Put\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000142263,\"lastEvaluation\":\"2024-04-12T22:36:27.142952901Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000914697,\"lastEvaluation\":\"2024-04-12T22:36:27.142183067Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imagestreams.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-62089bf6-dfc3-4319-af63-38d6191b31ea.yaml\",\"rules\":[{\"name\":\"imageregistry:imagestreamtags_count:sum\",\"query\":\"sum by (location, source) (image_registry_image_stream_tags_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000297936,\"lastEvaluation\":\"2024-04-12T22:36:29.517918454Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000312126,\"lastEvaluation\":\"2024-04-12T22:36:29.517908274Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress-to-route-controller.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-dfd4b459-b669-489c-9020-22defa539ee2.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"IngressWithoutClassName\",\"query\":\"openshift_ingress_to_route_controller_ingress_without_class_name == 1\",\"duration\":86400,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"alerts\":[{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.130.0.46:8443\",\"job\":\"route-controller-manager\",\"name\":\"test\",\"namespace\":\"e2e-test-router-stress-g64j9\",\"pod\":\"route-controller-manager-5bdb9f48f4-nbc4h\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress test is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:21:56.656877779Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.00057098,\"lastEvaluation\":\"2024-04-12T22:36:26.658168642Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UnmanagedRoutes\",\"query\":\"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is a Route owned by an unmanaged Ingress.\",\"message\":\"Route {{ $labels.name }} is owned by an unmanaged Ingress.\",\"summary\":\"Route owned by an Ingress no longer managed\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00055756,\"lastEvaluation\":\"2024-04-12T22:36:26.658742192Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001142721,\"lastEvaluation\":\"2024-04-12T22:36:26.658159241Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-dfd4b459-b669-489c-9020-22defa539ee2.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HAProxyDown\",\"query\":\"haproxy_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"This alert fires when metrics report that HAProxy is down.\",\"message\":\"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}\",\"summary\":\"HAProxy is down\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000117152,\"lastEvaluation\":\"2024-04-12T22:36:36.029997778Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HAProxyReloadFail\",\"query\":\"template_router_reload_failure == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.\",\"message\":\"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes\",\"summary\":\"HAProxy reload failure\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000192924,\"lastEvaluation\":\"2024-04-12T22:36:36.029803404Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerDegraded\",\"query\":\"ingress_controller_conditions{condition=\\\"Degraded\\\"} == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController status is degraded.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\ndegraded: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is degraded\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000116962,\"lastEvaluation\":\"2024-04-12T22:36:36.0301163Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerUnavailable\",\"query\":\"ingress_controller_conditions{condition=\\\"Available\\\"} == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController is not available.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\nunavailable: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is unavailable\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000086392,\"lastEvaluation\":\"2024-04-12T22:36:36.030234183Z\",\"type\":\"alerting\"},{\"name\":\"cluster:openshift_route_info:tls_termination:sum\",\"query\":\"sum by (tls_termination) (openshift_route_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000177153,\"lastEvaluation\":\"2024-04-12T22:36:36.030713762Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:avg\",\"query\":\"avg(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000077671,\"lastEvaluation\":\"2024-04-12T22:36:36.030548629Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:max\",\"query\":\"max(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000088982,\"lastEvaluation\":\"2024-04-12T22:36:36.030458617Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:median\",\"query\":\"quantile(0.5, route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000085012,\"lastEvaluation\":\"2024-04-12T22:36:36.0306273Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:min\",\"query\":\"min(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000134313,\"lastEvaluation\":\"2024-04-12T22:36:36.030321674Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001380943,\"lastEvaluation\":\"2024-04-12T22:36:36.028664144Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"insights\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-9ee17981-1562-4044-be51-c623d1ec6f10.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"InsightsDisabled\",\"query\":\"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\\\"Disabled\\\",name=\\\"insights\\\"} == 1)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html\",\"summary\":\"Insights operator is disabled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000255843,\"lastEvaluation\":\"2024-04-12T22:36:26.980475055Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"InsightsRecommendationActive\",\"query\":\"insights_recommendation_active == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights recommendation \\\"{{ $labels.description }}\\\" with total risk \\\"{{ $labels.total_risk }}\\\" was detected on the cluster. More information is available at {{ $labels.info_link }}.\",\"summary\":\"An Insights recommendation is active for this cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077884,\"lastEvaluation\":\"2024-04-12T22:36:26.980878496Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SimpleContentAccessNotAvailable\",\"query\":\"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\\\"SCAAvailable\\\",name=\\\"insights\\\",reason=\\\"NotFound\\\"}[5m]) == 0)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.\",\"summary\":\"Simple content access certificates are not available.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000144768,\"lastEvaluation\":\"2024-04-12T22:36:26.980732438Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000492636,\"lastEvaluation\":\"2024-04-12T22:36:26.980465834Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pre-release-lifecycle\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-0d5178dd-4e23-49ca-884b-5d9c83f14fb4.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"APIRemovedInNextEUSReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.29\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next EUS version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.059891486,\"lastEvaluation\":\"2024-04-12T22:36:49.753094018Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"APIRemovedInNextReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.29\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.057387063,\"lastEvaluation\":\"2024-04-12T22:36:49.695703445Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.117296501,\"lastEvaluation\":\"2024-04-12T22:36:49.695692644Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-audit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-3528d707-66c1-425c-bbdb-459ae3d79e76.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AuditLogError\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\\\".+-apiserver\\\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\\\".+-apiserver\\\"}[5m])) \\u003e 0\",\"duration\":60,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"An API Server had an error writing to an audit log.\",\"summary\":\"An API Server instance was unable to write audit logs. This could be\\ntriggered by the node running out of space, or a malicious actor\\ntampering with the audit logs.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000601002,\"lastEvaluation\":\"2024-04-12T22:36:35.38840949Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00049801,\"lastEvaluation\":\"2024-04-12T22:36:35.387886089Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"control-plane-cpu-utilization\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-7d5403fb-3bd5-4f24-ada6-427ad465fbde.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000343136,\"lastEvaluation\":\"2024-04-12T22:36:36.635727642Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000291696,\"lastEvaluation\":\"2024-04-12T22:36:36.635435026Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneCPU\",\"query\":\"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / count(kube_node_role{role=\\\"master\\\"}) \\u003e 60\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000682083,\"lastEvaluation\":\"2024-04-12T22:36:36.634751593Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001333476,\"lastEvaluation\":\"2024-04-12T22:36:36.634740183Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"api-performance\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-60c107f0-5c52-40c2-b772-4473cfac1e20.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000227105,\"lastEvaluation\":\"2024-04-12T22:36:54.152295946Z\",\"type\":\"recording\"},{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000176183,\"lastEvaluation\":\"2024-04-12T22:36:54.152524251Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.045000573,\"lastEvaluation\":\"2024-04-12T22:36:53.810314816Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.04588809,\"lastEvaluation\":\"2024-04-12T22:36:53.855324249Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:1m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000426938,\"lastEvaluation\":\"2024-04-12T22:36:53.809526111Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:5m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000356677,\"lastEvaluation\":\"2024-04-12T22:36:53.809955289Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003394156,\"lastEvaluation\":\"2024-04-12T22:36:54.131202038Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003666881,\"lastEvaluation\":\"2024-04-12T22:36:54.134598423Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000639063,\"lastEvaluation\":\"2024-04-12T22:36:54.124481797Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000439359,\"lastEvaluation\":\"2024-04-12T22:36:54.125588749Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002956427,\"lastEvaluation\":\"2024-04-12T22:36:54.119568202Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001952648,\"lastEvaluation\":\"2024-04-12T22:36:54.122527549Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005841684,\"lastEvaluation\":\"2024-04-12T22:36:54.077450485Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005037797,\"lastEvaluation\":\"2024-04-12T22:36:54.083295749Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.015395089,\"lastEvaluation\":\"2024-04-12T22:36:54.088746644Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.015411169,\"lastEvaluation\":\"2024-04-12T22:36:54.104151323Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000234615,\"lastEvaluation\":\"2024-04-12T22:36:54.088337676Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000170883,\"lastEvaluation\":\"2024-04-12T22:36:54.088574531Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005023357,\"lastEvaluation\":\"2024-04-12T22:36:54.067724007Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004696581,\"lastEvaluation\":\"2024-04-12T22:36:54.072750794Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003263723,\"lastEvaluation\":\"2024-04-12T22:36:54.143502616Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00255298,\"lastEvaluation\":\"2024-04-12T22:36:54.146768459Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002705812,\"lastEvaluation\":\"2024-04-12T22:36:54.138268044Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002524289,\"lastEvaluation\":\"2024-04-12T22:36:54.140976037Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002969107,\"lastEvaluation\":\"2024-04-12T22:36:54.149323749Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000371917,\"lastEvaluation\":\"2024-04-12T22:36:53.808903419Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000246655,\"lastEvaluation\":\"2024-04-12T22:36:53.809277166Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0046555,\"lastEvaluation\":\"2024-04-12T22:36:53.998565946Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.044223107,\"lastEvaluation\":\"2024-04-12T22:36:53.901220849Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.047218576,\"lastEvaluation\":\"2024-04-12T22:36:53.945452386Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.015738895,\"lastEvaluation\":\"2024-04-12T22:36:54.003766217Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.016407038,\"lastEvaluation\":\"2024-04-12T22:36:54.019513082Z\",\"type\":\"recording\"},{\"name\":\"instance_request_kind:apiserver_current_inflight_requests:sum\",\"query\":\"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000538041,\"lastEvaluation\":\"2024-04-12T22:36:54.003225456Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.091631305,\"lastEvaluation\":\"2024-04-12T22:36:51.904027503Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.102604299,\"lastEvaluation\":\"2024-04-12T22:36:51.996960149Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.32751045,\"lastEvaluation\":\"2024-04-12T22:36:52.725709768Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.56068537,\"lastEvaluation\":\"2024-04-12T22:36:53.053226938Z\",\"type\":\"recording\"},{\"name\":\"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum\",\"query\":\"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00046454,\"lastEvaluation\":\"2024-04-12T22:36:54.12512288Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.018830355,\"lastEvaluation\":\"2024-04-12T22:36:53.737931523Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.019527649,\"lastEvaluation\":\"2024-04-12T22:36:53.756770698Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005881424,\"lastEvaluation\":\"2024-04-12T22:36:53.992679242Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_storage_objects:max\",\"query\":\"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002836445,\"lastEvaluation\":\"2024-04-12T22:36:54.152701434Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000346107,\"lastEvaluation\":\"2024-04-12T22:36:54.130565845Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000287345,\"lastEvaluation\":\"2024-04-12T22:36:54.130913382Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.283265351,\"lastEvaluation\":\"2024-04-12T22:36:51.386615443Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.234128367,\"lastEvaluation\":\"2024-04-12T22:36:51.669889875Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.06807681,\"lastEvaluation\":\"2024-04-12T22:36:53.613921278Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.055914504,\"lastEvaluation\":\"2024-04-12T22:36:53.682007589Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.017559071,\"lastEvaluation\":\"2024-04-12T22:36:54.03592846Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.014222636,\"lastEvaluation\":\"2024-04-12T22:36:54.053495741Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002399647,\"lastEvaluation\":\"2024-04-12T22:36:54.126029407Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002131721,\"lastEvaluation\":\"2024-04-12T22:36:54.128431954Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.202137845,\"lastEvaluation\":\"2024-04-12T22:36:52.287428993Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.213206304,\"lastEvaluation\":\"2024-04-12T22:36:52.512496584Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.090115927,\"lastEvaluation\":\"2024-04-12T22:36:52.099571859Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.093912566,\"lastEvaluation\":\"2024-04-12T22:36:52.193510476Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0160272,\"lastEvaluation\":\"2024-04-12T22:36:53.776307217Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.016552801,\"lastEvaluation\":\"2024-04-12T22:36:53.792341868Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":2.769699168,\"lastEvaluation\":\"2024-04-12T22:36:51.385841661Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-requests-in-flight\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-6824c540-c854-48f2-9d56-c5eabf4a2b2f.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m\",\"query\":\"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver\\\"})[2m:])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000474705,\"lastEvaluation\":\"2024-04-12T22:36:27.870854345Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000483439,\"lastEvaluation\":\"2024-04-12T22:36:27.869890856Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-basic\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4f08eea7-5a77-4739-ba5f-4ebfa6a5c384.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1h) \\u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \\u003e (14.4 * 0.01)\",\"duration\":120,\"labels\":{\"long\":\"1h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"5m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000372347,\"lastEvaluation\":\"2024-04-12T22:36:25.801155645Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate6h) \\u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \\u003e (6 * 0.01)\",\"duration\":900,\"labels\":{\"long\":\"6h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"30m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000186373,\"lastEvaluation\":\"2024-04-12T22:36:25.801529542Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00057379,\"lastEvaluation\":\"2024-04-12T22:36:25.801145764Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4f08eea7-5a77-4739-ba5f-4ebfa6a5c384.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.015164194,\"lastEvaluation\":\"2024-04-12T22:36:38.558555321Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.010181167,\"lastEvaluation\":\"2024-04-12T22:36:38.588315118Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.009728989,\"lastEvaluation\":\"2024-04-12T22:36:38.548823922Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.007657447,\"lastEvaluation\":\"2024-04-12T22:36:38.598549263Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.010208408,\"lastEvaluation\":\"2024-04-12T22:36:38.538611644Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.005851462,\"lastEvaluation\":\"2024-04-12T22:36:38.606210031Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.014587073,\"lastEvaluation\":\"2024-04-12T22:36:38.573724735Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.009160747,\"lastEvaluation\":\"2024-04-12T22:36:38.612063442Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.121687879,\"lastEvaluation\":\"2024-04-12T22:36:39.048674253Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.130384218,\"lastEvaluation\":\"2024-04-12T22:36:38.918282635Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.123482134,\"lastEvaluation\":\"2024-04-12T22:36:38.794796241Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.083649567,\"lastEvaluation\":\"2024-04-12T22:36:38.637102042Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.073791771,\"lastEvaluation\":\"2024-04-12T22:36:38.72098503Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.008557175,\"lastEvaluation\":\"2024-04-12T22:36:38.621226979Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.007312438,\"lastEvaluation\":\"2024-04-12T22:36:38.629786694Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.619208356,\"lastEvaluation\":\"2024-04-12T22:36:38.537515729Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-extended\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-168fb4ca-48ed-4fa6-95a7-20c6c46562ef.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1d) \\u003e (3 * 0.01) and sum(apiserver_request:burnrate2h) \\u003e (3 * 0.01)\",\"duration\":3600,\"labels\":{\"long\":\"1d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"2h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000321587,\"lastEvaluation\":\"2024-04-12T22:36:35.100389895Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate3d) \\u003e (1 * 0.01) and sum(apiserver_request:burnrate6h) \\u003e (1 * 0.01)\",\"duration\":10800,\"labels\":{\"long\":\"3d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"6h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000424182,\"lastEvaluation\":\"2024-04-12T22:36:35.10077552Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.0004815,\"lastEvaluation\":\"2024-04-12T22:36:35.100382635Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-168fb4ca-48ed-4fa6-95a7-20c6c46562ef.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.019031292,\"lastEvaluation\":\"2024-04-12T22:36:31.518244181Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.010686038,\"lastEvaluation\":\"2024-04-12T22:36:31.553264243Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.022732989,\"lastEvaluation\":\"2024-04-12T22:36:31.495505362Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.012240721,\"lastEvaluation\":\"2024-04-12T22:36:31.563953511Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.015979289,\"lastEvaluation\":\"2024-04-12T22:36:31.537281113Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.011007505,\"lastEvaluation\":\"2024-04-12T22:36:31.576198032Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.083410281,\"lastEvaluation\":\"2024-04-12T22:36:31.495424416Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-5005c833-ea1b-4666-ae08-3bea362dc2d3.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TechPreviewNoUpgrade\",\"query\":\"cluster_feature_set{name!=\\\"\\\",namespace=\\\"openshift-kube-apiserver-operator\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.\",\"summary\":\"Cluster has enabled tech preview features that will prevent upgrades.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002395806,\"lastEvaluation\":\"2024-04-12T22:36:45.48300113Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002406596,\"lastEvaluation\":\"2024-04-12T22:36:45.48299372Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pod-security-violation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-be42b924-fa88-4cdc-b097-1e5e3015f2ef.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace!=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created in namespace \\\"{{ $labels.ocp_namespace }}\\\" but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000136227,\"lastEvaluation\":\"2024-04-12T22:36:27.163648332Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads users created in the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000253205,\"lastEvaluation\":\"2024-04-12T22:36:27.163347548Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000473305,\"lastEvaluation\":\"2024-04-12T22:36:27.163313064Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-59a05266-d32f-408a-b2c9-134a0aa47a1f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"GarbageCollectorSyncFailed\",\"query\":\"rate(garbagecollector_controller_resources_sync_error_total[5m]) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md\",\"summary\":\"There was a problem with syncing the resources for garbage collection.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000191034,\"lastEvaluation\":\"2024-04-12T22:36:26.687631783Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeControllerManagerDown\",\"query\":\"absent(up{job=\\\"kube-controller-manager\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-controller-manager\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeControllerManager has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000311836,\"lastEvaluation\":\"2024-04-12T22:36:26.686338099Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetAtLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \\u003e 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md\",\"summary\":\"The pod disruption budget is preventing further disruption to pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000466764,\"lastEvaluation\":\"2024-04-12T22:36:26.686652636Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \\u003c kube_poddisruptionbudget_status_desired_healthy)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md\",\"summary\":\"The pod disruption budget registers insufficient amount of pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000365197,\"lastEvaluation\":\"2024-04-12T22:36:26.687264646Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00114217,\"lastEvaluation\":\"2024-04-12T22:36:26.686326939Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-14fb49f3-f1fb-4874-93f5-bc12c6437bcf.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeSchedulerDown\",\"query\":\"absent(up{job=\\\"scheduler\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-scheduler\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeScheduler has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000321926,\"lastEvaluation\":\"2024-04-12T22:36:31.339316817Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000278595,\"lastEvaluation\":\"2024-04-12T22:36:31.338463854Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"scheduler-legacy-policy-deprecated\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-14fb49f3-f1fb-4874-93f5-bc12c6437bcf.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SchedulerLegacyPolicySet\",\"query\":\"cluster_legacy_scheduler_policy \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.\",\"summary\":\"Legacy scheduler policy API in use by the scheduler.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000216224,\"lastEvaluation\":\"2024-04-12T22:36:43.443324055Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00019135,\"lastEvaluation\":\"2024-04-12T22:36:43.442976787Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-api-operator-metrics-collector-up\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineAPIOperatorMetricsCollectionFailing\",\"query\":\"mapi_mao_collector_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"For more details:  oc logs \\u003cmachine-api-operator-pod-name\\u003e -n openshift-machine-api\",\"summary\":\"machine api operator metrics collection is failing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000266805,\"lastEvaluation\":\"2024-04-12T22:36:53.975148282Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.0001851,\"lastEvaluation\":\"2024-04-12T22:36:53.972384944Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-health-check-unterminated-short-circuit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineHealthCheckUnterminatedShortCircuit\",\"query\":\"mapi_machinehealthcheck_short_circuit == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\\nthe status of machines in the cluster.\\n\",\"summary\":\"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000221112,\"lastEvaluation\":\"2024-04-12T22:36:51.928398519Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000250995,\"lastEvaluation\":\"2024-04-12T22:36:51.927690636Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-not-yet-deleted\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineNotYetDeleted\",\"query\":\"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\\\"Deleting\\\"}[15m])) \\u003e 0\",\"duration\":21600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine is not properly deleting, this may be due to a configuration issue with the\\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\\nlong termination periods which are preventing deletion.\\n\",\"summary\":\"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000272524,\"lastEvaluation\":\"2024-04-12T22:36:44.941617306Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000309326,\"lastEvaluation\":\"2024-04-12T22:36:44.940729497Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-with-no-running-phase\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithNoRunningPhase\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\\\"Running|Deleting\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine has been without a Running or Deleting phase for more than 60 minutes.\\nThe machine may not have been provisioned properly from the infrastructure provider, or\\nit might have issues with CertificateSigningRequests being approved.\\n\",\"summary\":\"machine {{ $labels.name }} is in phase: {{ $labels.phase }}\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000326337,\"lastEvaluation\":\"2024-04-12T22:36:31.397113273Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000339177,\"lastEvaluation\":\"2024-04-12T22:36:31.397104472Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-without-valid-node-ref\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithoutValidNode\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"If the machine never became a node, you should diagnose the machine related failures.\\nIf the node was deleted from the API, you may delete the machine if appropriate.\\n\",\"summary\":\"machine {{ $labels.name }} does not have valid node reference\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000335148,\"lastEvaluation\":\"2024-04-12T22:36:45.843108201Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000343898,\"lastEvaluation\":\"2024-04-12T22:36:45.843102441Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-drain-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCDrainError\",\"query\":\"mcc_drain_err \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000240684,\"lastEvaluation\":\"2024-04-12T22:36:26.126555741Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000255873,\"lastEvaluation\":\"2024-04-12T22:36:26.125837504Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-pool-alert\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCPoolAlert\",\"query\":\"mcc_pool_alert \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000245005,\"lastEvaluation\":\"2024-04-12T22:36:45.350111834Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000216412,\"lastEvaluation\":\"2024-04-12T22:36:45.34879619Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"os-image-override.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"name\":\"os_image_url_override:sum\",\"query\":\"sum(os_image_url_override)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000269855,\"lastEvaluation\":\"2024-04-12T22:36:25.675348321Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000282795,\"lastEvaluation\":\"2024-04-12T22:36:25.675338511Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"extremely-high-individual-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneMemory\",\"query\":\"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 90\",\"duration\":2700,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.\",\"summary\":\"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000650663,\"lastEvaluation\":\"2024-04-12T22:36:39.133743062Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000662383,\"lastEvaluation\":\"2024-04-12T22:36:39.133734082Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"high-overall-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneMemory\",\"query\":\"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 60\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.\",\"summary\":\"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000830794,\"lastEvaluation\":\"2024-04-12T22:36:37.080318668Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000843374,\"lastEvaluation\":\"2024-04-12T22:36:37.080309428Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-kubelet-health-state-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeletHealthState\",\"query\":\"mcd_kubelet_state \\u003e 2\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet health failure threshold reached\",\"summary\":\"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000346499,\"lastEvaluation\":\"2024-04-12T22:36:36.713249675Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000359399,\"lastEvaluation\":\"2024-04-12T22:36:36.713240175Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-pivot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDPivotError\",\"query\":\"mcd_pivot_errors_total \\u003e 0\",\"duration\":120,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000243424,\"lastEvaluation\":\"2024-04-12T22:36:26.054822042Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000284765,\"lastEvaluation\":\"2024-04-12T22:36:26.05464721Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-reboot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDRebootError\",\"query\":\"mcd_reboots_failed_total \\u003e 0\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000254155,\"lastEvaluation\":\"2024-04-12T22:36:54.372174709Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000257584,\"lastEvaluation\":\"2024-04-12T22:36:54.37207428Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"system-memory-exceeds-reservation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SystemMemoryExceedsReservation\",\"query\":\"sum by (node) (container_memory_rss{id=\\\"/system.slice\\\"}) \\u003e ((sum by (node) (kube_node_status_capacity{resource=\\\"memory\\\"} - kube_node_status_allocatable{resource=\\\"memory\\\"})) * 0.95)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).\",\"summary\":\"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000574741,\"lastEvaluation\":\"2024-04-12T22:36:44.345946605Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000587701,\"lastEvaluation\":\"2024-04-12T22:36:44.345936865Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"operator.marketplace.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-marketplace-marketplace-alert-rules-714df196-fc7d-484a-8308-076cc71239c1.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"OperatorHubSourceError\",\"query\":\"catalogsource_ready{exported_namespace=\\\"openshift-marketplace\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Operators shipped via the {{ $labels.name }} source are not available for installation until the issue is fixed. Operators already installed from this source will not receive updates until issue is fixed. Inspect the status of the pod owned by {{ $labels.name }} source in the openshift-marketplace namespace (oc -n openshift-marketplace get pods -l olm.catalogSource={{ $labels.name }}) to diagnose and repair.\",\"summary\":\"The {{ $labels.name }} source is in non-ready state for more than 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000205474,\"lastEvaluation\":\"2024-04-12T22:36:26.137690192Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000410352,\"lastEvaluation\":\"2024-04-12T22:36:26.137019818Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"alertmanager.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-alertmanager-main-rules-b89ef7fc-8e25-4a61-90d5-f3ea1f8d7983.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterDown\",\"query\":\"(count by (namespace, service) (avg_over_time(up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c 0.5) / count by (namespace, service) (up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) \\u003e= 0.5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of Alertmanager instances within the {{$labels.job}} cluster have been up for less than half of the last 5m.\",\"summary\":\"Half or more of the Alertmanager instances within the same cluster are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000275936,\"lastEvaluation\":\"2024-04-12T22:36:40.32919809Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterFailedToSendAlerts\",\"query\":\"min by (namespace, service, integration) (rate(alertmanager_notifications_failed_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The minimum notification failure rate to {{ $labels.integration }} sent from any instance in the {{$labels.job}} cluster is {{ $value | humanizePercentage }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerClusterFailedToSendAlerts.md\",\"summary\":\"All Alertmanager instances in a cluster failed to send notifications to a critical integration.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000824656,\"lastEvaluation\":\"2024-04-12T22:36:40.328170879Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerConfigInconsistent\",\"query\":\"count by (namespace, service) (count_values by (namespace, service) (\\\"config_hash\\\", alertmanager_config_hash{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) != 1\",\"duration\":1200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager instances within the {{$labels.job}} cluster have different configurations.\",\"summary\":\"Alertmanager instances within the same cluster have different configurations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000199453,\"lastEvaluation\":\"2024-04-12T22:36:40.328996846Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedReload\",\"query\":\"max_over_time(alertmanager_config_last_reload_successful{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Configuration has failed to load for {{ $labels.namespace }}/{{ $labels.pod}}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedReload.md\",\"summary\":\"Reloading an Alertmanager configuration has failed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000355179,\"lastEvaluation\":\"2024-04-12T22:36:40.326475027Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedToSendAlerts\",\"query\":\"(rate(alertmanager_notifications_failed_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} failed to send {{ $value | humanizePercentage }} of notifications to {{ $labels.integration }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedToSendAlerts.md\",\"summary\":\"An Alertmanager instance failed to send notifications.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000700037,\"lastEvaluation\":\"2024-04-12T22:36:40.327184894Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerMembersInconsistent\",\"query\":\"max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c on (namespace, service) group_left () count by (namespace, service) (max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]))\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} has only found {{ $value }} members of the {{$labels.job}} cluster.\",\"summary\":\"A member of an Alertmanager cluster has not found all other cluster members.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000351658,\"lastEvaluation\":\"2024-04-12T22:36:40.326831955Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002459311,\"lastEvaluation\":\"2024-04-12T22:36:40.326465486Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-list-watch.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"apiserver_list_watch_request_success_total:rate:sum\",\"query\":\"sum by (verb) (rate(apiserver_request_total{code=~\\\"2..\\\",verb=~\\\"LIST|WATCH\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011091991,\"lastEvaluation\":\"2024-04-12T22:36:49.03612487Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.011121632,\"lastEvaluation\":\"2024-04-12T22:36:49.036099009Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"Watchdog\",\"query\":\"vector(1)\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"alerts\":[{\"labels\":{\"alertname\":\"Watchdog\",\"namespace\":\"openshift-monitoring\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"state\":\"firing\",\"activeAt\":\"2024-04-12T21:38:00.04805628Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.00036572,\"lastEvaluation\":\"2024-04-12T22:36:30.049373349Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000398011,\"lastEvaluation\":\"2024-04-12T22:36:30.049344508Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"count:up0\",\"query\":\"count without (instance, pod, node) (up == 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000584401,\"lastEvaluation\":\"2024-04-12T22:36:30.562035441Z\",\"type\":\"recording\"},{\"name\":\"count:up1\",\"query\":\"count without (instance, pod, node) (up == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001188013,\"lastEvaluation\":\"2024-04-12T22:36:30.560845548Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001916136,\"lastEvaluation\":\"2024-04-12T22:36:30.560597722Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-node-recording.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:node_cpu:ratio\",\"query\":\"cluster:node_cpu:sum_rate5m / count(sum by (instance, cpu) (node_cpu_seconds_total))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001001909,\"lastEvaluation\":\"2024-04-12T22:36:31.717688757Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:sum_rate5m\",\"query\":\"sum(rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000501319,\"lastEvaluation\":\"2024-04-12T22:36:31.717186508Z\",\"type\":\"recording\"},{\"name\":\"instance:node_cpu:rate:sum\",\"query\":\"sum by (instance) (rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000744524,\"lastEvaluation\":\"2024-04-12T22:36:31.715888793Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_receive_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000298946,\"lastEvaluation\":\"2024-04-12T22:36:31.716635227Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_transmit_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_transmit_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000250275,\"lastEvaluation\":\"2024-04-12T22:36:31.716935283Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002753246,\"lastEvaluation\":\"2024-04-12T22:36:31.71556568Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-recurring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:usage:workload:capacity_physical_cpu_core_seconds\",\"query\":\"sum_over_time(workload:capacity_physical_cpu_cores:sum[30s:1s]) + ((cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) or (absent(cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) * 0))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000427389,\"lastEvaluation\":\"2024-04-12T22:36:34.377895604Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.00037001,\"lastEvaluation\":\"2024-04-12T22:36:34.377729995Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-network\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeNetworkInterfaceFlapping\",\"query\":\"changes(node_network_up{device!~\\\"veth.+|tunbr\\\",job=\\\"node-exporter\\\"}[2m]) \\u003e 2\",\"duration\":120,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Network interface \\\"{{ $labels.device }}\\\" changing its up status often on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\",\"summary\":\"Network interface is often changing its status\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000479155,\"lastEvaluation\":\"2024-04-12T22:36:25.834301237Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000490206,\"lastEvaluation\":\"2024-04-12T22:36:25.834294496Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-build.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"openshift:build_by_strategy:sum\",\"query\":\"sum by (strategy) (openshift_build_status_phase_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000352997,\"lastEvaluation\":\"2024-04-12T22:36:39.546313771Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000482146,\"lastEvaluation\":\"2024-04-12T22:36:39.545145852Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd-telemetry.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000471499,\"lastEvaluation\":\"2024-04-12T22:36:46.87614077Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000474385,\"lastEvaluation\":\"2024-04-12T22:36:46.874714269Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000240762,\"lastEvaluation\":\"2024-04-12T22:36:46.874471787Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_use_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000215515,\"lastEvaluation\":\"2024-04-12T22:36:46.875923125Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000687606,\"lastEvaluation\":\"2024-04-12T22:36:46.875190305Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002108802,\"lastEvaluation\":\"2024-04-12T22:36:46.874461786Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TargetDown\",\"query\":\"100 * ((1 - sum by (job, namespace, service) (up and on (namespace, pod) kube_pod_info) / count by (job, namespace, service) (up and on (namespace, pod) kube_pod_info)) or (count by (job, namespace, service) (up == 0) / count by (job, namespace, service) (up))) \\u003e 10\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ printf \\\"%.4g\\\" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.\",\"summary\":\"Some targets were not reachable from the monitoring server for an extended period of time.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00725452,\"lastEvaluation\":\"2024-04-12T22:36:49.131392564Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.007818947,\"lastEvaluation\":\"2024-04-12T22:36:49.131316966Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:ingress_controller_aws_nlb_active:sum\",\"query\":\"sum(ingress_controller_aws_nlb_active) or vector(0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000117253,\"lastEvaluation\":\"2024-04-12T22:36:47.632569255Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_in:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_in_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000150232,\"lastEvaluation\":\"2024-04-12T22:36:47.628183781Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_out:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_out_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000159203,\"lastEvaluation\":\"2024-04-12T22:36:47.628335493Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_connections:sum\",\"query\":\"sum(haproxy_frontend_current_sessions)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000108811,\"lastEvaluation\":\"2024-04-12T22:36:47.628495827Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace=~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00157298,\"lastEvaluation\":\"2024-04-12T22:36:47.629900964Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001092682,\"lastEvaluation\":\"2024-04-12T22:36:47.631475284Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace!~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000785795,\"lastEvaluation\":\"2024-04-12T22:36:47.628605789Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00050599,\"lastEvaluation\":\"2024-04-12T22:36:47.629393424Z\",\"type\":\"recording\"},{\"name\":\"code:cluster:ingress_http_request_count:rate5m:sum\",\"query\":\"sum by (code) (rate(haproxy_server_http_responses_total[5m]) \\u003e 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001585241,\"lastEvaluation\":\"2024-04-12T22:36:47.62659673Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006338168,\"lastEvaluation\":\"2024-04-12T22:36:47.610707779Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-kubernetes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"AlertmanagerReceiversNotConfigured\",\"query\":\"cluster:alertmanager_integrations:max == 0\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"alerts\":[{\"labels\":{\"alertname\":\"AlertmanagerReceiversNotConfigured\",\"namespace\":\"openshift-monitoring\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"state\":\"firing\",\"activeAt\":\"2024-04-12T21:39:17.894182481Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000165894,\"lastEvaluation\":\"2024-04-12T22:36:47.996157584Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterMonitoringOperatorReconciliationErrors\",\"query\":\"max_over_time(cluster_monitoring_operator_last_reconciliation_successful[5m]) == 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors are occurring during reconciliation cycles. Inspect the cluster-monitoring-operator log for potential root causes.\",\"summary\":\"Cluster Monitoring Operator is experiencing unexpected reconciliation errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077241,\"lastEvaluation\":\"2024-04-12T22:36:47.996079603Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentReplicasMismatch\",\"query\":\"(((kube_deployment_spec_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_deployment_status_replicas_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_deployment_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)) * on () group_left () cluster:control_plane:all_nodes_ready) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes. This indicates that cluster infrastructure is unable to start or restart the necessary components. This most often occurs when one or more nodes are down or partioned from the cluster, or a fault occurs on the node that prevents the workload from starting. In rare cases this may indicate a new version of a cluster component cannot start due to a bug or configuration error. Assess the pods for this deployment to verify they are running on healthy nodes and then contact support.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeDeploymentReplicasMismatch.md\",\"summary\":\"Deployment has not matched the expected number of replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002321535,\"lastEvaluation\":\"2024-04-12T22:36:47.996324498Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodNotScheduled\",\"query\":\"last_over_time(kube_pod_status_unschedulable{namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} cannot be scheduled for more than 30 minutes.\\nCheck the details of the pod with the following command:\\noc describe -n {{ $labels.namespace }} pod {{ $labels.pod }}\",\"summary\":\"Pod cannot be scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000541001,\"lastEvaluation\":\"2024-04-12T22:36:48.025027584Z\",\"type\":\"alerting\"},{\"name\":\"cluster:alertmanager_integrations:max\",\"query\":\"max(alertmanager_integrations{namespace=\\\"openshift-monitoring\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000121752,\"lastEvaluation\":\"2024-04-12T22:36:47.995154745Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (label_replace(cluster:infra_nodes, \\\"label_node_role_kubernetes_io\\\", \\\"infra\\\", \\\"\\\", \\\"\\\") * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000464569,\"lastEvaluation\":\"2024-04-12T22:36:47.964889868Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000130143,\"lastEvaluation\":\"2024-04-12T22:36:47.966423848Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_sockets_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (max by (node, package, label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000144733,\"lastEvaluation\":\"2024-04-12T22:36:47.994840189Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_memory_bytes:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000365347,\"lastEvaluation\":\"2024-04-12T22:36:47.966555301Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_cpu_usage:ratio\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m])) / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008544355,\"lastEvaluation\":\"2024-04-12T22:36:47.952716206Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_spec_cpu_shares:ratio\",\"query\":\"sum(container_spec_cpu_shares{container=\\\"\\\",pod!=\\\"\\\"}) / 1000 / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008869612,\"lastEvaluation\":\"2024-04-12T22:36:47.943845054Z\",\"type\":\"recording\"},{\"name\":\"cluster:control_plane:all_nodes_ready\",\"query\":\"sum(min by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}) and max by (node) (kube_node_role{role=\\\"master\\\"})) == bool sum(kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000271916,\"lastEvaluation\":\"2024-04-12T22:36:47.995738736Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_hyperthreading\",\"query\":\"clamp_max(label_replace(sum by (instance, package, core) (node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_replace(label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003e 1, \\\"label_node_hyperthread_enabled\\\", \\\"true\\\", \\\"instance\\\", \\\"(.*)\\\") or on (instance, package) label_replace(sum by (instance, package, core) (label_replace(node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003c= 1, \\\"label_node_hyperthread_enabled\\\", \\\"false\\\", \\\"instance\\\", \\\"(.*)\\\"), 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000775525,\"lastEvaluation\":\"2024-04-12T22:36:47.965356087Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_node_labels\",\"query\":\"topk by (node) (1, cluster:nodes_roles) * on (node) group_right (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_node_openshift_io_os_id, label_kubernetes_io_arch, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) label_replace(cluster:cpu_core_hyperthreading, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000288696,\"lastEvaluation\":\"2024-04-12T22:36:47.966133642Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_usage_cores:sum\",\"query\":\"sum(1 - rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[2m]) * on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{pod=~\\\"node-exporter.+\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00054354,\"lastEvaluation\":\"2024-04-12T22:36:47.966921808Z\",\"type\":\"recording\"},{\"name\":\"cluster:hyperthread_enabled_nodes\",\"query\":\"kube_node_labels and on (node) (sum by (node, package, core) (label_replace(node_cpu_info, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")) == 2)\",\"labels\":{\"label_node_hyperthread_enabled\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000349047,\"lastEvaluation\":\"2024-04-12T22:36:47.964357608Z\",\"type\":\"recording\"},{\"name\":\"cluster:infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"infra\\\"})\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000170604,\"lastEvaluation\":\"2024-04-12T22:36:47.963852708Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolume_plugin_type_counts:sum\",\"query\":\"sum by (plugin_name, volume_mode) (pv_collector_total_pv_count{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000459179,\"lastEvaluation\":\"2024-04-12T22:36:47.995278127Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_resource_requests_storage_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000755684,\"lastEvaluation\":\"2024-04-12T22:36:47.985956067Z\",\"type\":\"recording\"},{\"name\":\"cluster:kubelet_volume_stats_used_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kubelet_volume_stats_used_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000401228,\"lastEvaluation\":\"2024-04-12T22:36:47.987359644Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (cluster:master_nodes and on (node) cluster:infra_nodes)\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000132182,\"lastEvaluation\":\"2024-04-12T22:36:47.964024441Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"label_node_role_kubernetes_io\":\"master\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000325696,\"lastEvaluation\":\"2024-04-12T22:36:47.963525342Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage:ratio\",\"query\":\"sum by (cluster) (container_memory_usage_bytes{container=\\\"\\\",pod!=\\\"\\\"}) / sum by (cluster) (machine_memory_bytes)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010134309,\"lastEvaluation\":\"2024-04-12T22:36:47.933709305Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage_bytes:sum\",\"query\":\"sum(node_memory_MemTotal_bytes{job=\\\"node-exporter\\\"} - node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000199454,\"lastEvaluation\":\"2024-04-12T22:36:47.967467268Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_instance_type_count:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) (cluster:master_nodes or on (node) kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000256065,\"lastEvaluation\":\"2024-04-12T22:36:47.985698522Z\",\"type\":\"recording\"},{\"name\":\"cluster:nodes_roles\",\"query\":\"cluster:master_infra_nodes or on (node) cluster:master_nodes or on (node) cluster:infra_nodes or on (node) max without (endpoint, instance, job, pod, service) (kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000198834,\"lastEvaluation\":\"2024-04-12T22:36:47.964157634Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:containers:sum\",\"query\":\"sum(max by (instance) (kubelet_containers_per_pod_count_sum))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000128573,\"lastEvaluation\":\"2024-04-12T22:36:47.994546103Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_node_ready:avg5m\",\"query\":\"avg_over_time((count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1)) / scalar(count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003232883,\"lastEvaluation\":\"2024-04-12T22:36:48.004967685Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_schedulable_node_ready_reachable:avg5m\",\"query\":\"avg_over_time((((count((max by (node) (up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1) and max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1) and min by (node) (kube_node_spec_unschedulable == 0))) / scalar(count(min by (node) (kube_node_spec_unschedulable == 0))))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006318392,\"lastEvaluation\":\"2024-04-12T22:36:47.998647403Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace=~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001252884,\"lastEvaluation\":\"2024-04-12T22:36:48.022030176Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:pods:terminal:workload:sum\",\"query\":\"count(count by (namespace, pod) (kube_pod_restart_policy{namespace!~\\\"openshift-.+\\\",type!=\\\"Always\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000127912,\"lastEvaluation\":\"2024-04-12T22:36:47.994417121Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:resources:sum\",\"query\":\"topk(500, max by (resource) (apiserver_storage_objects != -1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002758324,\"lastEvaluation\":\"2024-04-12T22:36:47.991656857Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:max:5m\",\"query\":\"max_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000062601,\"lastEvaluation\":\"2024-04-12T22:36:47.987296113Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:min:5m\",\"query\":\"min_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000095062,\"lastEvaluation\":\"2024-04-12T22:36:47.987200051Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace!~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001740804,\"lastEvaluation\":\"2024-04-12T22:36:48.02328452Z\",\"type\":\"recording\"},{\"name\":\"cluster:virt_platform_nodes:sum\",\"query\":\"count by (type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (sum by (instance, type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (virt_platform))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000180903,\"lastEvaluation\":\"2024-04-12T22:36:47.964707865Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_object_counts:sum\",\"query\":\"sum by (instance) (apiserver_storage_objects != -1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003892765,\"lastEvaluation\":\"2024-04-12T22:36:47.987761702Z\",\"type\":\"recording\"},{\"name\":\"kube_running_pod_ready\",\"query\":\"(max without (condition, container, endpoint, instance, job, service) (((kube_pod_status_ready{condition=\\\"false\\\"} == 1) * 0 or (kube_pod_status_ready{condition=\\\"true\\\"} == 1)) * on (pod, namespace) group_left () group by (pod, namespace) (kube_pod_status_phase{phase=~\\\"Running|Unknown|Pending\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013824628,\"lastEvaluation\":\"2024-04-12T22:36:48.008202298Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_cpu_usage:sum\",\"query\":\"sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=\\\"\\\",container!=\\\"POD\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007228534,\"lastEvaluation\":\"2024-04-12T22:36:47.92647816Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_memory_usage_bytes:sum\",\"query\":\"sum by (namespace) (container_memory_usage_bytes{container!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009425742,\"lastEvaluation\":\"2024-04-12T22:36:47.917049358Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_cores:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000163023,\"lastEvaluation\":\"2024-04-12T22:36:47.994675856Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_sockets:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (max by (node, package, label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000167663,\"lastEvaluation\":\"2024-04-12T22:36:47.994985912Z\",\"type\":\"recording\"},{\"name\":\"openshift:cpu_usage_cores:sum\",\"query\":\"cluster:cpu_usage_cores:sum - workload:cpu_usage_cores:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000148032,\"lastEvaluation\":\"2024-04-12T22:36:47.975989784Z\",\"type\":\"recording\"},{\"name\":\"openshift:memory_usage_bytes:sum\",\"query\":\"cluster:memory_usage_bytes:sum - workload:memory_usage_bytes:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000188714,\"lastEvaluation\":\"2024-04-12T22:36:47.985507768Z\",\"type\":\"recording\"},{\"name\":\"pod:container_cpu_usage:sum\",\"query\":\"sum by (pod, namespace) (rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010408544,\"lastEvaluation\":\"2024-04-12T22:36:47.895636169Z\",\"type\":\"recording\"},{\"name\":\"pod:container_fs_usage_bytes:sum\",\"query\":\"sum by (pod, namespace) (container_fs_usage_bytes{pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010997915,\"lastEvaluation\":\"2024-04-12T22:36:47.906047623Z\",\"type\":\"recording\"},{\"name\":\"profile:cluster_monitoring_operator_collection_profile:max\",\"query\":\"max by (profile) (cluster_monitoring_operator_collection_profile == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000066711,\"lastEvaluation\":\"2024-04-12T22:36:47.996011892Z\",\"type\":\"recording\"},{\"name\":\"workload:capacity_physical_cpu_cores:sum\",\"query\":\"(sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_infra=\\\"\\\",label_node_role_kubernetes_io_master=\\\"\\\"} or absent(__does_not_exist__) * 0)) + ((sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_master=\\\"true\\\"} or absent(__does_not_exist__) * 0) * ((max(cluster_master_schedulable == 1) * 0 + 1) or (absent(cluster_master_schedulable == 1) * 0))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00048553,\"lastEvaluation\":\"2024-04-12T22:36:47.986712971Z\",\"type\":\"recording\"},{\"name\":\"workload:cpu_usage_cores:sum\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008318751,\"lastEvaluation\":\"2024-04-12T22:36:47.967667942Z\",\"type\":\"recording\"},{\"name\":\"workload:memory_usage_bytes:sum\",\"query\":\"sum(container_memory_working_set_bytes{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009365742,\"lastEvaluation\":\"2024-04-12T22:36:47.976139526Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.127353787,\"lastEvaluation\":\"2024-04-12T22:36:47.895626938Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-monitoring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"monitoring:container_memory_working_set_bytes:sum\",\"query\":\"sum by (namespace) (max without (instance) (container_memory_working_set_bytes{container=\\\"\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000993363,\"lastEvaluation\":\"2024-04-12T22:36:30.768302406Z\",\"type\":\"recording\"},{\"name\":\"monitoring:haproxy_server_http_responses_total:sum\",\"query\":\"sum by (exported_service) (rate(haproxy_server_http_responses_total{exported_namespace=\\\"openshift-monitoring\\\",exported_service=~\\\"alertmanager-main|prometheus-k8s\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000488686,\"lastEvaluation\":\"2024-04-12T22:36:30.772285858Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_samples_post_metric_relabeling:topk3\",\"query\":\"topk(3, max by (namespace, job) (topk by (namespace, job) (1, scrape_samples_post_metric_relabeling)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00095385,\"lastEvaluation\":\"2024-04-12T22:36:30.771330297Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_series_added:topk3_sum1h\",\"query\":\"topk(3, sum by (namespace, job) (sum_over_time(scrape_series_added[1h])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002031528,\"lastEvaluation\":\"2024-04-12T22:36:30.769297129Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicationController\\\"}, \\\"replicationcontroller\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicationcontroller, namespace) group_left (owner_name) topk by (replicationcontroller, namespace) (1, max by (replicationcontroller, namespace, owner_name) (kube_replicationcontroller_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deploymentconfig\"},\"health\":\"ok\",\"evaluationTime\":0.000470985,\"lastEvaluation\":\"2024-04-12T22:36:30.772775774Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_samples_appended_total:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (rate(prometheus_tsdb_head_samples_appended_total{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[2m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000264694,\"lastEvaluation\":\"2024-04-12T22:36:30.768035902Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_series:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (prometheus_tsdb_head_series{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000344929,\"lastEvaluation\":\"2024-04-12T22:36:30.767688283Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.005570036,\"lastEvaluation\":\"2024-04-12T22:36:30.767678773Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-sre.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"code:apiserver_request_total:rate:sum\",\"query\":\"sum by (code) (rate(apiserver_request_total{job=\\\"apiserver\\\"}[10m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.019782155,\"lastEvaluation\":\"2024-04-12T22:36:31.411271461Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.019792586,\"lastEvaluation\":\"2024-04-12T22:36:31.41126545Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-vsphere.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:vsphere_csi_migration:max\",\"query\":\"max by (status) (vsphere_csi_migration{status=~\\\"|LegacyDeprecatedInTreeDriver|CSIWithMigrationDriver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000243123,\"lastEvaluation\":\"2024-04-12T22:36:26.80731085Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_esxi_version_total:sum\",\"query\":\"sum by (version) (vsphere_esxi_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000080114,\"lastEvaluation\":\"2024-04-12T22:36:26.807030775Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_infrastructure_failure_domains:max\",\"query\":\"max by (scope) (vsphere_infrastructure_failure_domains)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000066604,\"lastEvaluation\":\"2024-04-12T22:36:26.807243016Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_node_hw_version_total:sum\",\"query\":\"sum by (hw_version) (vsphere_node_hw_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000065644,\"lastEvaluation\":\"2024-04-12T22:36:26.807112469Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_topology_tags:max\",\"query\":\"max by (source) (vsphere_topology_tags)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000062223,\"lastEvaluation\":\"2024-04-12T22:36:26.807179523Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_vcenter_info:sum\",\"query\":\"sum by (version, build) (vsphere_vcenter_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000231123,\"lastEvaluation\":\"2024-04-12T22:36:26.806797263Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000583501,\"lastEvaluation\":\"2024-04-12T22:36:26.805968505Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-state-metrics\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kube-state-metrics-rules-07cb1c2d-024f-4864-ad8b-21ee1306da90.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeStateMetricsListErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in list operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in list operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000475439,\"lastEvaluation\":\"2024-04-12T22:36:54.944450425Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStateMetricsWatchErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in watch operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in watch operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000437549,\"lastEvaluation\":\"2024-04-12T22:36:54.944927434Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000924658,\"lastEvaluation\":\"2024-04-12T22:36:54.944443735Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_cpu_usage_seconds_total\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate\",\"query\":\"sum by (cluster, namespace, pod, container) (irate(container_cpu_usage_seconds_total{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"}[5m])) * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012626512,\"lastEvaluation\":\"2024-04-12T22:36:47.746380067Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.011551674,\"lastEvaluation\":\"2024-04-12T22:36:47.745823281Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_cache\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_cache\",\"query\":\"container_memory_cache{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010943622,\"lastEvaluation\":\"2024-04-12T22:36:35.623483087Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.010970132,\"lastEvaluation\":\"2024-04-12T22:36:35.623473857Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_rss\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_rss\",\"query\":\"container_memory_rss{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010789669,\"lastEvaluation\":\"2024-04-12T22:36:52.911826657Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.0108036,\"lastEvaluation\":\"2024-04-12T22:36:52.911817216Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_swap\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_swap\",\"query\":\"container_memory_swap{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011071578,\"lastEvaluation\":\"2024-04-12T22:36:38.910817703Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.01108283,\"lastEvaluation\":\"2024-04-12T22:36:38.910809572Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_working_set_bytes\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_working_set_bytes\",\"query\":\"container_memory_working_set_bytes{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012711706,\"lastEvaluation\":\"2024-04-12T22:36:50.154884787Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.011223838,\"lastEvaluation\":\"2024-04-12T22:36:50.154405196Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_resource\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004433776,\"lastEvaluation\":\"2024-04-12T22:36:42.149576059Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006873023,\"lastEvaluation\":\"2024-04-12T22:36:42.126663768Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004633216,\"lastEvaluation\":\"2024-04-12T22:36:42.140182459Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009029006,\"lastEvaluation\":\"2024-04-12T22:36:42.109512855Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005386616,\"lastEvaluation\":\"2024-04-12T22:36:42.154011685Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006560217,\"lastEvaluation\":\"2024-04-12T22:36:42.133539481Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004755714,\"lastEvaluation\":\"2024-04-12T22:36:42.144817696Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008116488,\"lastEvaluation\":\"2024-04-12T22:36:42.11854467Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.051555493,\"lastEvaluation\":\"2024-04-12T22:36:42.107845418Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.pod_owner\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"DaemonSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"daemonset\"},\"health\":\"ok\",\"evaluationTime\":0.001090018,\"lastEvaluation\":\"2024-04-12T22:36:38.838936232Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicaSet\\\"}, \\\"replicaset\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicaset, namespace) group_left (owner_name) topk by (replicaset, namespace) (1, max by (replicaset, namespace, owner_name) (kube_replicaset_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deployment\"},\"health\":\"ok\",\"evaluationTime\":0.002620491,\"lastEvaluation\":\"2024-04-12T22:36:38.835415538Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"Job\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"job\"},\"health\":\"ok\",\"evaluationTime\":0.000240513,\"lastEvaluation\":\"2024-04-12T22:36:38.840454482Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"StatefulSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"statefulset\"},\"health\":\"ok\",\"evaluationTime\":0.000424552,\"lastEvaluation\":\"2024-04-12T22:36:38.84002826Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.00409193,\"lastEvaluation\":\"2024-04-12T22:36:38.835407698Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-scheduler.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000126387,\"lastEvaluation\":\"2024-04-12T22:36:42.25654512Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000113555,\"lastEvaluation\":\"2024-04-12T22:36:42.255835723Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000120876,\"lastEvaluation\":\"2024-04-12T22:36:42.255011989Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000096905,\"lastEvaluation\":\"2024-04-12T22:36:42.255950319Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000138198,\"lastEvaluation\":\"2024-04-12T22:36:42.255134295Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000271785,\"lastEvaluation\":\"2024-04-12T22:36:42.254155403Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000494666,\"lastEvaluation\":\"2024-04-12T22:36:42.256048034Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000559229,\"lastEvaluation\":\"2024-04-12T22:36:42.255274063Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000580281,\"lastEvaluation\":\"2024-04-12T22:36:42.254429078Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.00205232,\"lastEvaluation\":\"2024-04-12T22:36:42.250809735Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubelet.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000744184,\"lastEvaluation\":\"2024-04-12T22:36:46.115656106Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000682296,\"lastEvaluation\":\"2024-04-12T22:36:46.114953035Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001092848,\"lastEvaluation\":\"2024-04-12T22:36:46.113858227Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.00259969,\"lastEvaluation\":\"2024-04-12T22:36:46.1138033Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-apps\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"KubeContainerWaiting\",\"query\":\"sum by (namespace, pod, container, cluster) (kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"pod/{{ $labels.pod }} in namespace {{ $labels.namespace }} on container {{ $labels.container}} has been in waiting state for longer than 1 hour.\",\"summary\":\"Pod container waiting longer than 1 hour\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubeContainerWaiting\",\"container\":\"node-probe\",\"namespace\":\"openshift-must-gather-dxffw\",\"pod\":\"perf-node-gather-daemonset-b5xc5\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"pod/perf-node-gather-daemonset-b5xc5 in namespace openshift-must-gather-dxffw on container node-probe has been in waiting state for longer than 1 hour.\",\"summary\":\"Pod container waiting longer than 1 hour\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000662295,\"lastEvaluation\":\"2024-04-12T22:36:28.236250899Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetMisScheduled\",\"query\":\"kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.\",\"summary\":\"DaemonSet pods are misscheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000344848,\"lastEvaluation\":\"2024-04-12T22:36:28.237572849Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetNotScheduled\",\"query\":\"kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} - kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.\",\"summary\":\"DaemonSet pods are not scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000657115,\"lastEvaluation\":\"2024-04-12T22:36:28.236914854Z\",\"type\":\"alerting\"},{\"state\":\"pending\",\"name\":\"KubeDaemonSetRolloutStuck\",\"query\":\"((kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != 0) or (kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} has not finished or progressed for at least 30 minutes.\",\"summary\":\"DaemonSet rollout is stuck.\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubeDaemonSetRolloutStuck\",\"container\":\"kube-rbac-proxy-main\",\"daemonset\":\"perf-node-gather-daemonset\",\"endpoint\":\"https-main\",\"job\":\"kube-state-metrics\",\"namespace\":\"openshift-must-gather-dxffw\",\"service\":\"kube-state-metrics\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"DaemonSet openshift-must-gather-dxffw/perf-node-gather-daemonset has not finished or progressed for at least 30 minutes.\",\"summary\":\"DaemonSet rollout is stuck.\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.002392867,\"lastEvaluation\":\"2024-04-12T22:36:28.233856691Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentGenerationMismatch\",\"query\":\"kube_deployment_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_deployment_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.\",\"summary\":\"Deployment generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001193103,\"lastEvaluation\":\"2024-04-12T22:36:28.228656795Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentRolloutStuck\",\"query\":\"kube_deployment_status_condition{condition=\\\"Progressing\\\",job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",status=\\\"false\\\"} != 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Rollout of deployment {{ $labels.namespace }}/{{ $labels.deployment }} is not progressing for longer than 15 minutes.\",\"summary\":\"Deployment rollout is not progressing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000670776,\"lastEvaluation\":\"2024-04-12T22:36:28.229851058Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaMaxedOut\",\"query\":\"kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has been running at max replicas for longer than 15 minutes.\",\"summary\":\"HPA is running at max replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000459619,\"lastEvaluation\":\"2024-04-12T22:36:28.24056117Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaReplicasMismatch\",\"query\":\"(kube_horizontalpodautoscaler_status_desired_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_horizontalpodautoscaler_spec_min_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003c kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and changes(kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[15m]) == 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has not matched the desired number of replicas for longer than 15 minutes.\",\"summary\":\"HPA has not matched desired number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001479378,\"lastEvaluation\":\"2024-04-12T22:36:28.239080822Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobFailed\",\"query\":\"kube_job_failed{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeJobFailed.md\",\"summary\":\"Job failed to complete.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000314316,\"lastEvaluation\":\"2024-04-12T22:36:28.238765466Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobNotCompleted\",\"query\":\"time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} and kube_job_status_active{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0) \\u003e 43200\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than {{ \\\"43200\\\" | humanizeDuration }} to complete.\",\"summary\":\"Job did not complete in time\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000697987,\"lastEvaluation\":\"2024-04-12T22:36:28.237918717Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodCrashLooping\",\"query\":\"max_over_time(kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",reason=\\\"CrashLoopBackOff\\\"}[5m]) \\u003e= 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is in waiting state (reason: \\\"CrashLoopBackOff\\\").\",\"summary\":\"Pod is crash looping.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000842105,\"lastEvaluation\":\"2024-04-12T22:36:28.220530281Z\",\"type\":\"alerting\"},{\"state\":\"pending\",\"name\":\"KubePodNotReady\",\"query\":\"sum by (namespace, pod, cluster) (max by (namespace, pod, cluster) (kube_pod_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Pending|Unknown\\\"} unless ignoring (phase) (kube_pod_status_unschedulable{job=\\\"kube-state-metrics\\\"} == 1)) * on (namespace, pod, cluster) group_left (owner_kind) topk by (namespace, pod, cluster) (1, max by (namespace, pod, owner_kind, cluster) (kube_pod_owner{owner_kind!=\\\"Job\\\"}))) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePodNotReady.md\",\"summary\":\"Pod has been in a non-ready state for more than 15 minutes.\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubePodNotReady\",\"namespace\":\"openshift-must-gather-dxffw\",\"pod\":\"perf-node-gather-daemonset-b5xc5\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod openshift-must-gather-dxffw/perf-node-gather-daemonset-b5xc5 has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePodNotReady.md\",\"summary\":\"Pod has been in a non-ready state for more than 15 minutes.\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.007222574,\"lastEvaluation\":\"2024-04-12T22:36:28.221432431Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetGenerationMismatch\",\"query\":\"kube_statefulset_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.\",\"summary\":\"StatefulSet generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000705964,\"lastEvaluation\":\"2024-04-12T22:36:28.231651031Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetReplicasMismatch\",\"query\":\"(kube_statefulset_status_replicas_ready{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[10m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.\",\"summary\":\"StatefulSet has not matched the expected number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001109779,\"lastEvaluation\":\"2024-04-12T22:36:28.230522894Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetUpdateNotRolledOut\",\"query\":\"(max without (revision) (kube_statefulset_status_current_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} unless kube_statefulset_status_update_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) * (kube_statefulset_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.\",\"summary\":\"StatefulSet update has not been rolled out.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00162914,\"lastEvaluation\":\"2024-04-12T22:36:28.232358075Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.020502179,\"lastEvaluation\":\"2024-04-12T22:36:28.22052125Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-resources\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeCPUOvercommit\",\"query\":\"sum by (cluster) (namespace_cpu:kube_pod_container_resource_requests:sum{job=\\\"kube-state-metrics\\\"}) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted CPU resource requests for Pods by {{ $value }} CPU shares and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted CPU resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558299,\"lastEvaluation\":\"2024-04-12T22:36:38.436962344Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeMemoryOvercommit\",\"query\":\"sum by (cluster) (namespace_memory:kube_pod_container_resource_requests:sum) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted memory resource requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted memory resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558559,\"lastEvaluation\":\"2024-04-12T22:36:38.437522304Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaAlmostFull\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) \\u003e 0.9 \\u003c 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota is going to be full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001061567,\"lastEvaluation\":\"2024-04-12T22:36:38.438082153Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaExceeded\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) \\u003e 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota has exceeded the limits.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000732379,\"lastEvaluation\":\"2024-04-12T22:36:38.439862808Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaFullyUsed\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota is fully used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000716728,\"lastEvaluation\":\"2024-04-12T22:36:38.43914473Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003457577,\"lastEvaluation\":\"2024-04-12T22:36:38.431377955Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-storage\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeErrors\",\"query\":\"kube_persistentvolume_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Failed|Pending\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The persistent volume {{ $labels.persistentvolume }} has status {{ $labels.phase }}.\",\"summary\":\"PersistentVolume is having issues with provisioning.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000957141,\"lastEvaluation\":\"2024-04-12T22:36:33.212259431Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeFillingUp\",\"query\":\"(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_capacity_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.03 and kubelet_volume_stats_used_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage }} free.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeFillingUp.md\",\"summary\":\"PersistentVolume is filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001723601,\"lastEvaluation\":\"2024-04-12T22:36:33.206464493Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeFillingUp\",\"query\":\"(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_capacity_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.15 and kubelet_volume_stats_used_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 and predict_linear(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[6h], 4 * 24 * 3600) \\u003c 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to fill up within four days. Currently {{ $value | humanizePercentage }} is available.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeFillingUp.md\",\"summary\":\"PersistentVolume is filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001571234,\"lastEvaluation\":\"2024-04-12T22:36:33.208189584Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeInodesFillingUp\",\"query\":\"(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_inodes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.03 and kubelet_volume_stats_inodes_used{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} only has {{ $value | humanizePercentage }} free inodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeInodesFillingUp.md\",\"summary\":\"PersistentVolumeInodes are filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001091778,\"lastEvaluation\":\"2024-04-12T22:36:33.209761908Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeInodesFillingUp\",\"query\":\"(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_inodes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.15 and kubelet_volume_stats_inodes_used{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 and predict_linear(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[6h], 4 * 24 * 3600) \\u003c 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to run out of inodes within four days. Currently {{ $value | humanizePercentage }} of its inodes are free.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeInodesFillingUp.md\",\"summary\":\"PersistentVolumeInodes are filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001403395,\"lastEvaluation\":\"2024-04-12T22:36:33.210854846Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.007493132,\"lastEvaluation\":\"2024-04-12T22:36:33.20553579Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeClientErrors\",\"query\":\"(sum by (cluster, instance, job, namespace) (rate(rest_client_requests_total{code=~\\\"5..\\\",job=\\\"apiserver\\\"}[5m])) / sum by (cluster, instance, job, namespace) (rate(rest_client_requests_total{job=\\\"apiserver\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ $value | humanizePercentage }} errors.'\",\"summary\":\"Kubernetes API server client is experiencing errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000975762,\"lastEvaluation\":\"2024-04-12T22:36:46.739292704Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000988642,\"lastEvaluation\":\"2024-04-12T22:36:46.739282454Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system-apiserver\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIDown\",\"query\":\"absent(up{job=\\\"apiserver\\\"} == 1)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeAPI has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeAPIDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000153003,\"lastEvaluation\":\"2024-04-12T22:36:36.883777021Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPITerminatedRequests\",\"query\":\"sum(rate(apiserver_request_terminations_total{job=\\\"apiserver\\\"}[10m])) / (sum(rate(apiserver_request_total{job=\\\"apiserver\\\"}[10m])) + sum(rate(apiserver_request_terminations_total{job=\\\"apiserver\\\"}[10m]))) \\u003e 0.2\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.\",\"summary\":\"The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.021009137,\"lastEvaluation\":\"2024-04-12T22:36:36.883945044Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAggregatedAPIDown\",\"query\":\"(1 - max by (name, namespace, cluster) (avg_over_time(aggregator_unavailable_apiservice{job=\\\"apiserver\\\"}[10m]))) * 100 \\u003c 85\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m.\",\"summary\":\"Kubernetes aggregated API is down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001569211,\"lastEvaluation\":\"2024-04-12T22:36:36.88220667Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAggregatedAPIErrors\",\"query\":\"sum by (name, namespace, cluster) (increase(aggregator_unavailable_apiservice_total{job=\\\"apiserver\\\"}[10m])) \\u003e 4\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has reported errors. It has appeared unavailable {{ $value | humanize }} times averaged over the past 10m.\",\"summary\":\"Kubernetes aggregated API has reported errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000228844,\"lastEvaluation\":\"2024-04-12T22:36:36.881976606Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.026801515,\"lastEvaluation\":\"2024-04-12T22:36:36.881571853Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system-kubelet\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeNodeNotReady\",\"query\":\"kube_node_status_condition{condition=\\\"Ready\\\",job=\\\"kube-state-metrics\\\",status=\\\"true\\\"} == 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.node }} has been unready for more than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeNodeNotReady.md\",\"summary\":\"Node is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000347466,\"lastEvaluation\":\"2024-04-12T22:36:47.477697163Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeNodeReadinessFlapping\",\"query\":\"sum by (cluster, node) (changes(kube_node_status_condition{condition=\\\"Ready\\\",job=\\\"kube-state-metrics\\\",status=\\\"true\\\"}[15m])) \\u003e 2\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The readiness status of node {{ $labels.node }} has changed {{ $value }} times in the last 15 minutes.\",\"summary\":\"Node readiness status is flapping.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000338697,\"lastEvaluation\":\"2024-04-12T22:36:47.485972463Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeNodeUnreachable\",\"query\":\"(kube_node_spec_taint{effect=\\\"NoSchedule\\\",job=\\\"kube-state-metrics\\\",key=\\\"node.kubernetes.io/unreachable\\\"} unless ignoring (key, value) kube_node_spec_taint{job=\\\"kube-state-metrics\\\",key=~\\\"ToBeDeletedByClusterAutoscaler|cloud.google.com/impending-node-termination|aws-node-termination-handler/spot-itn\\\"}) == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.node }} is unreachable and some workloads may be rescheduled.\",\"summary\":\"Node is unreachable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000345126,\"lastEvaluation\":\"2024-04-12T22:36:47.47804645Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletClientCertificateRenewalErrors\",\"query\":\"increase(kubelet_certificate_manager_client_expiration_renew_errors[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet on node {{ $labels.node }} has failed to renew its client certificate ({{ $value | humanize }} errors in the last 5 minutes).\",\"summary\":\"Kubelet has failed to renew its client certificate.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000166143,\"lastEvaluation\":\"2024-04-12T22:36:47.489302558Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletDown\",\"query\":\"absent(up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Kubelet has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeletDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000211714,\"lastEvaluation\":\"2024-04-12T22:36:47.489627074Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletPlegDurationHigh\",\"query\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile{quantile=\\\"0.99\\\"} \\u003e= 10\",\"duration\":300,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Kubelet Pod Lifecycle Event Generator has a 99th percentile duration of {{ $value }} seconds on node {{ $labels.node }}.\",\"summary\":\"Kubelet Pod Lifecycle Event Generator is taking too long to relist.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000133302,\"lastEvaluation\":\"2024-04-12T22:36:47.48631348Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletPodStartUpLatencyHigh\",\"query\":\"histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m]))) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} \\u003e 60\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on node {{ $labels.node }}.\",\"summary\":\"Kubelet Pod startup latency is too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002851685,\"lastEvaluation\":\"2024-04-12T22:36:47.486448053Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletServerCertificateRenewalErrors\",\"query\":\"increase(kubelet_server_expiration_renew_errors[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet on node {{ $labels.node }} has failed to renew its server certificate ({{ $value | humanize }} errors in the last 5 minutes).\",\"summary\":\"Kubelet has failed to renew its server certificate.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000155903,\"lastEvaluation\":\"2024-04-12T22:36:47.489469911Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletTooManyPods\",\"query\":\"count by (cluster, node) ((kube_pod_status_phase{job=\\\"kube-state-metrics\\\",phase=\\\"Running\\\"} == 1) * on (instance, pod, namespace, cluster) group_left (node) topk by (instance, pod, namespace, cluster) (1, kube_pod_info{job=\\\"kube-state-metrics\\\"})) / max by (cluster, node) (kube_node_status_capacity{job=\\\"kube-state-metrics\\\",resource=\\\"pods\\\"} != 1) \\u003e 0.95\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage }} of its Pod capacity.\",\"summary\":\"Kubelet is running at capacity.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.007577107,\"lastEvaluation\":\"2024-04-12T22:36:47.478392976Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.010869158,\"lastEvaluation\":\"2024-04-12T22:36:47.47747049Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\":node_memory_MemAvailable_bytes:sum\",\"query\":\"sum by (cluster) (node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"} or (node_memory_Buffers_bytes{job=\\\"node-exporter\\\"} + node_memory_Cached_bytes{job=\\\"node-exporter\\\"} + node_memory_MemFree_bytes{job=\\\"node-exporter\\\"} + node_memory_Slab_bytes{job=\\\"node-exporter\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000403148,\"lastEvaluation\":\"2024-04-12T22:36:46.186635932Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:ratio_rate5m\",\"query\":\"avg by (cluster) (node:node_cpu_utilization:ratio_rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000116852,\"lastEvaluation\":\"2024-04-12T22:36:46.187712183Z\",\"type\":\"recording\"},{\"name\":\"node:node_cpu_utilization:ratio_rate5m\",\"query\":\"avg by (cluster, node) (sum without (mode) (rate(node_cpu_seconds_total{job=\\\"node-exporter\\\",mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000670913,\"lastEvaluation\":\"2024-04-12T22:36:46.18704021Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod:kube_pod_info:\",\"query\":\"topk by (cluster, namespace, pod) (1, max by (cluster, node, namespace, pod) (label_replace(kube_pod_info{job=\\\"kube-state-metrics\\\",node!=\\\"\\\"}, \\\"pod\\\", \\\"$1\\\", \\\"pod\\\", \\\"(.*)\\\")))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005129449,\"lastEvaluation\":\"2024-04-12T22:36:46.181502733Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006371299,\"lastEvaluation\":\"2024-04-12T22:36:46.181250843Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-exporter\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-node-exporter-rules-9b822991-f20a-4cb2-be29-fcb975430f9b.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeBondingDegraded\",\"query\":\"(node_bonding_slaves - node_bonding_active) != 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Bonding interface {{ $labels.master }} on {{ $labels.instance }} is in degraded state due to one or more slave failures.\",\"summary\":\"Bonding interface is degraded\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000116593,\"lastEvaluation\":\"2024-04-12T22:36:35.574705821Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeClockNotSynchronising\",\"query\":\"(min_over_time(node_timex_sync_status{job=\\\"node-exporter\\\"}[5m]) == 0 and node_timex_maxerror_seconds{job=\\\"node-exporter\\\"} \\u003e= 16) and on () absent(up{job=\\\"ptp-monitor-service\\\"})\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Clock at {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeClockNotSynchronising.md\",\"summary\":\"Clock not synchronising.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000225155,\"lastEvaluation\":\"2024-04-12T22:36:35.573165571Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeClockSkewDetected\",\"query\":\"((node_timex_offset_seconds{job=\\\"node-exporter\\\"} \\u003e 0.05 and deriv(node_timex_offset_seconds{job=\\\"node-exporter\\\"}[5m]) \\u003e= 0) or (node_timex_offset_seconds{job=\\\"node-exporter\\\"} \\u003c -0.05 and deriv(node_timex_offset_seconds{job=\\\"node-exporter\\\"}[5m]) \\u003c= 0)) and on () absent(up{job=\\\"ptp-monitor-service\\\"})\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Clock at {{ $labels.instance }} is out of sync by more than 0.05s. Ensure NTP is configured correctly on this host.\",\"summary\":\"Clock skew detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000332756,\"lastEvaluation\":\"2024-04-12T22:36:35.572831975Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFileDescriptorLimit\",\"query\":\"(node_filefd_allocated{job=\\\"node-exporter\\\"} * 100 / node_filefd_maximum{job=\\\"node-exporter\\\"} \\u003e 90)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"File descriptors limit at {{ $labels.instance }} is currently at {{ printf \\\"%.2f\\\" $value }}%.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFileDescriptorLimit.md\",\"summary\":\"Kernel is predicted to exhaust file descriptors limit soon.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000181364,\"lastEvaluation\":\"2024-04-12T22:36:35.573972417Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFileDescriptorLimit\",\"query\":\"(node_filefd_allocated{job=\\\"node-exporter\\\"} * 100 / node_filefd_maximum{job=\\\"node-exporter\\\"} \\u003e 70)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"File descriptors limit at {{ $labels.instance }} is currently at {{ printf \\\"%.2f\\\" $value }}%.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFileDescriptorLimit.md\",\"summary\":\"Kernel is predicted to exhaust file descriptors limit soon.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000214924,\"lastEvaluation\":\"2024-04-12T22:36:35.573756543Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfFiles\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 3 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfFiles.md\",\"summary\":\"Filesystem has less than 3% inodes left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00465352,\"lastEvaluation\":\"2024-04-12T22:36:35.566829759Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfFiles\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 5 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfFiles.md\",\"summary\":\"Filesystem has less than 5% inodes left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00460984,\"lastEvaluation\":\"2024-04-12T22:36:35.562218109Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfSpace\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 3 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfSpace.md\",\"summary\":\"Filesystem has less than 3% space left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00414662,\"lastEvaluation\":\"2024-04-12T22:36:35.540116821Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfSpace\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 5 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfSpace.md\",\"summary\":\"Filesystem has less than 5% space left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.004205861,\"lastEvaluation\":\"2024-04-12T22:36:35.535909749Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemFilesFillingUp\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 20 and predict_linear(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 4 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left and is filling up fast.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemFilesFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of inodes within the next 4 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.008696209,\"lastEvaluation\":\"2024-04-12T22:36:35.55351983Z\",\"type\":\"alerting\"},{\"state\":\"inacti
Ginkgo exit error 1: exit with code 1

# [sig-instrumentation][Late] OpenShift alerting rules [apigroup:image.openshift.io] should have a valid severity label [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:108]: Failed to fetch alerting rules: unable to parse response "{\"status\":\"success\",\"data\":{\"groups\":[{\"name\":\"CloudCredentialOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-59f29d20-14e3-45ac-95b2-817cbe434ddc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorDeprovisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsDeprovisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be cleaned up\",\"summary\":\"One or more CredentialsRequest CRs are unable to be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000117926,\"lastEvaluation\":\"2024-04-12T22:36:31.735304389Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorInsufficientCloudCreds\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"InsufficientCloudCreds\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.\",\"message\":\"Cluster's cloud credentials insufficient for minting or passthrough\",\"summary\":\"Problem with the available platform credentials.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000108965,\"lastEvaluation\":\"2024-04-12T22:36:31.735423516Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorProvisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsProvisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be fulfilled\",\"summary\":\"One or more CredentialsRequest CRs are unable to be processed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000098825,\"lastEvaluation\":\"2024-04-12T22:36:31.735204354Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorStaleCredentials\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"StaleCredentials\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.\",\"message\":\"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).\",\"summary\":\"One or more CredentialsRequest CRs are stale and should be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000105386,\"lastEvaluation\":\"2024-04-12T22:36:31.735533862Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorTargetNamespaceMissing\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"MissingTargetNamespace\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.\",\"message\":\"CredentialsRequest(s) pointing to non-existent namespace\",\"summary\":\"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000150643,\"lastEvaluation\":\"2024-04-12T22:36:31.735039966Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00053507,\"lastEvaluation\":\"2024-04-12T22:36:31.735037066Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-machine-approver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-02d4c4c8-9481-4482-a4eb-aa2196fb1abd.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineApproverMaxPendingCSRsReached\",\"query\":\"mapi_current_pending_csr \\u003e mapi_max_pending_csr\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of pending CertificateSigningRequests has exceeded the\\nmaximum threshold (current number of machine + 100). Check the\\npending CSRs to determine which machines need approval, also check\\nthat the nodelink controller is running in the openshift-machine-api\\nnamespace.\\n\",\"summary\":\"max pending CSRs threshold reached.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000264985,\"lastEvaluation\":\"2024-04-12T22:36:35.040054675Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000294385,\"lastEvaluation\":\"2024-04-12T22:36:35.039883304Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-tuning-operator.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-475016e5-fc0a-4af1-bd38-a45d26c2b85a.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NTODegraded\",\"query\":\"nto_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Node Tuning Operator is degraded. Review the \\\"node-tuning\\\" ClusterOperator object for further details.\",\"summary\":\"The Node Tuning Operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000087741,\"lastEvaluation\":\"2024-04-12T22:36:35.756300122Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NTOPodsNotReady\",\"query\":\"kube_pod_status_ready{condition=\\\"true\\\",namespace=\\\"openshift-cluster-node-tuning-operator\\\"} == 0\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.pod }} is not ready.\\nReview the \\\"Event\\\" objects in \\\"openshift-cluster-node-tuning-operator\\\" namespace for further details.\\n\",\"summary\":\"Pod {{ $labels.pod }} is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000291586,\"lastEvaluation\":\"2024-04-12T22:36:35.756006506Z\",\"type\":\"alerting\"},{\"name\":\"nto_custom_profiles:count\",\"query\":\"count by (_id) (nto_profile_calculated_total{profile!~\\\"openshift\\\",profile!~\\\"openshift-control-plane\\\",profile!~\\\"openshift-node\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000113402,\"lastEvaluation\":\"2024-04-12T22:36:35.756391654Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000503047,\"lastEvaluation\":\"2024-04-12T22:36:35.755518805Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"SamplesOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-5c7cc52f-a9cb-47fe-a845-20e9f9a26649.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SamplesDegraded\",\"query\":\"openshift_samples_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples could not be deployed and the operator is degraded. Review the \\\"openshift-samples\\\" ClusterOperator object for further details.\\n\",\"summary\":\"Samples operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000093025,\"lastEvaluation\":\"2024-04-12T22:36:47.104513835Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesImagestreamImportFailing\",\"query\":\"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \\u003e sum(openshift_samples_failed_imagestream_import_info)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports.  You can look at the \\\"openshift-samples\\\"\\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is detecting problems with imagestream image imports\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000264614,\"lastEvaluation\":\"2024-04-12T22:36:47.10424756Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesInvalidConfig\",\"query\":\"openshift_samples_invalidconfig_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator has been given an invalid configuration.\\n\",\"summary\":\"Samples operator Invalid configuration\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077034,\"lastEvaluation\":\"2024-04-12T22:36:47.10460794Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingSecret\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_secret\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator cannot find the samples pull secret in the openshift namespace.\\n\",\"summary\":\"Samples operator is not able to find secret\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000078804,\"lastEvaluation\":\"2024-04-12T22:36:47.104685694Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingTBRCredential\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_tbr_credential\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\\n\",\"summary\":\"Samples operator is not able to find the credentials for registry\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000072294,\"lastEvaluation\":\"2024-04-12T22:36:47.104765228Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesRetriesMissingOnImagestreamImportFailing\",\"query\":\"sum(openshift_samples_failed_imagestream_import_info) \\u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\\nimports are not occurring.  Contact support.  You can look at the \\\"openshift-samples\\\" ClusterOperator object\\nfor details. Most likely there are issues with the external image registry hosting the images that need to\\nbe investigated.  The list of ImageStreams that have failing imports are:\\n{{ range query \\\"openshift_samples_failed_imagestream_import_info \\u003e 0\\\" }}\\n  {{ .Labels.name }}\\n{{ end }}\\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is having problems with imagestream imports and its retries.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000430863,\"lastEvaluation\":\"2024-04-12T22:36:47.103814857Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesTBRInaccessibleOnBoot\",\"query\":\"openshift_samples_tbr_inaccessible_info == 1\",\"duration\":172800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"One of two situations has occurred.  Either\\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\\nassociated with each image stream tag from the samples catalog is\\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\\nassist the mirroring process.\\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\\n\",\"summary\":\"Samples operator is not able to access the registry on boot\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000079234,\"lastEvaluation\":\"2024-04-12T22:36:47.104838392Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001115289,\"lastEvaluation\":\"2024-04-12T22:36:47.103804797Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"default-storage-classes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MultipleDefaultStorageClasses\",\"query\":\"max_over_time(default_storage_class_count[5m]) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster storage operator monitors all storage classes configured in the cluster\\nand checks there is not more than one default StorageClass configured.\\n\",\"message\":\"StorageClass count check is failing (there should not be more than one default StorageClass)\",\"summary\":\"More than one default StorageClass detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000251395,\"lastEvaluation\":\"2024-04-12T22:36:37.477974261Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000378831,\"lastEvaluation\":\"2024-04-12T22:36:37.477148942Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodStartupStorageOperationsFailing\",\"query\":\"increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status!=\\\"success\\\"}[5m]) \\u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status=\\\"success\\\"}[5m]) == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Failing storage operation \\\"{{ $labels.operation_name }}\\\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\\nfrom starting for past 5 minutes.\\nPlease investigate Pods that are \\\"ContainerCreating\\\" on the node: \\\"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\\\".\\nEvents of the Pods should contain exact error message: \\\"oc describe pod -n \\u003cpod namespace\\u003e \\u003cpod name\\u003e\\\".\\n\",\"summary\":\"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003662945,\"lastEvaluation\":\"2024-04-12T22:36:49.473150763Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003676175,\"lastEvaluation\":\"2024-04-12T22:36:49.473141023Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-selinux.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"name\":\"cluster:volume_manager_selinux_pod_context_mismatch_total\",\"query\":\"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000381297,\"lastEvaluation\":\"2024-04-12T22:36:28.076271377Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volume_context_mismatch_errors_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000109212,\"lastEvaluation\":\"2024-04-12T22:36:28.076799817Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000142513,\"lastEvaluation\":\"2024-04-12T22:36:28.076655824Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volumes_admitted_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000286085,\"lastEvaluation\":\"2024-04-12T22:36:28.076910049Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001153521,\"lastEvaluation\":\"2024-04-12T22:36:28.075904055Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-operators\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-b7ec8a0e-ca37-48ff-91bd-48b44cd606d6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotEvaluateConditionalUpdates\",\"query\":\"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\\\"Recommended\\\",status=\\\"Unknown\\\"}) \\u003e= 3600)\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.\",\"summary\":\"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000149303,\"lastEvaluation\":\"2024-04-12T22:36:25.502645639Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterNotUpgradeable\",\"query\":\"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\\\"Upgradeable\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"Upgradeable\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0) (ne (len (label \\\"reason\\\" $value)) 0) }}{{label \\\"reason\\\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000282415,\"lastEvaluation\":\"2024-04-12T22:36:25.50115166Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDegraded\",\"query\":\"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\\\"Degraded\\\",job=\\\"cluster-version-operator\\\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\\\"cluster-version-operator\\\"})) == 1)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has been degraded for 30 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000644752,\"lastEvaluation\":\"2024-04-12T22:36:25.501743902Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDown\",\"query\":\"max by (namespace, name, reason) (cluster_operator_up{job=\\\"cluster-version-operator\\\"} == 0)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has not been available for 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000306316,\"lastEvaluation\":\"2024-04-12T22:36:25.501435936Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorFlapping\",\"query\":\"max by (namespace, name) (changes(cluster_operator_up{job=\\\"cluster-version-operator\\\"}[2m]) \\u003e 2)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator up status is changing often.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000254825,\"lastEvaluation\":\"2024-04-12T22:36:25.502389944Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001656412,\"lastEvaluation\":\"2024-04-12T22:36:25.50114171Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-b7ec8a0e-ca37-48ff-91bd-48b44cd606d6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotRetrieveUpdates\",\"query\":\"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \\u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\\\"RetrievedUpdates\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\",reason!=\\\"NoChannel\\\"}))\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"RetrievedUpdates\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0)}}{{label \\\"reason\\\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\\\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\\\n'}\\\"`{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000241142,\"lastEvaluation\":\"2024-04-12T22:36:39.739623191Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterReleaseNotAccepted\",\"query\":\"max by (namespace, name, reason) (cluster_operator_conditions{condition=\\\"ReleaseAccepted\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"The desired cluster release has not been accepted for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000125806,\"lastEvaluation\":\"2024-04-12T22:36:39.73996714Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterVersionOperatorDown\",\"query\":\"absent(up{job=\\\"cluster-version-operator\\\"} == 1)\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-cluster-version\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} For more information refer to {{ label \\\"url\\\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}\",\"summary\":\"Cluster version operator has disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000237812,\"lastEvaluation\":\"2024-04-12T22:36:39.739383739Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UpdateAvailable\",\"query\":\"sum by (channel, namespace, upstream) (cluster_version_available_updates) \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Your upstream update recommendation service recommends you update your cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000098335,\"lastEvaluation\":\"2024-04-12T22:36:39.739867784Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000646182,\"lastEvaluation\":\"2024-04-12T22:36:39.738991737Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift/console-operator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-13a53b5a-afd9-4211-9680-fe10e32351a7.yaml\",\"rules\":[{\"name\":\"cluster:console_auth_login_failures_total:sum\",\"query\":\"sum by (reason) (console_auth_login_failures_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000116322,\"lastEvaluation\":\"2024-04-12T22:36:29.799875967Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_requests_total:sum\",\"query\":\"sum(console_auth_login_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000191814,\"lastEvaluation\":\"2024-04-12T22:36:29.799544241Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_successes_total:sum\",\"query\":\"sum by (role) (console_auth_login_successes_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000133083,\"lastEvaluation\":\"2024-04-12T22:36:29.799738254Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_logout_requests_total:sum\",\"query\":\"sum by (reason) (console_auth_logout_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000080572,\"lastEvaluation\":\"2024-04-12T22:36:29.799993909Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_customization_perspectives_info:max\",\"query\":\"max by (name, state) (console_customization_perspectives_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000078672,\"lastEvaluation\":\"2024-04-12T22:36:29.800314685Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_plugins_info:max\",\"query\":\"max by (name, state) (console_plugins_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000163453,\"lastEvaluation\":\"2024-04-12T22:36:29.800149772Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_usage_users:max\",\"query\":\"max by (role) (console_usage_users)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000072861,\"lastEvaluation\":\"2024-04-12T22:36:29.800076061Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000865197,\"lastEvaluation\":\"2024-04-12T22:36:29.79953109Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-dns.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-be119138-53ca-458e-a33e-816e88c6ca38.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CoreDNSErrorsHigh\",\"query\":\"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\\\"SERVFAIL\\\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.\",\"summary\":\"CoreDNS serverfail\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000269855,\"lastEvaluation\":\"2024-04-12T22:36:51.98821851Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSHealthCheckSlow\",\"query\":\"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \\u003e 10\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})\",\"summary\":\"CoreDNS health checks\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000361857,\"lastEvaluation\":\"2024-04-12T22:36:51.987855743Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSPanicking\",\"query\":\"increase(coredns_panics_total[10m]) \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}\",\"summary\":\"CoreDNS panic\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000244395,\"lastEvaluation\":\"2024-04-12T22:36:51.987609808Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000887197,\"lastEvaluation\":\"2024-04-12T22:36:51.987603338Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"etcd\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-bdafb11d-bf7f-4480-ae4e-7cbeddeea773.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdDatabaseHighFragmentationRatio\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \\u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \\u003e 1.048576e+08\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.\",\"runbook_url\":\"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation\",\"summary\":\"etcd database size in use is less than 50% of the actual allocated storage.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000273924,\"lastEvaluation\":\"2024-04-12T22:36:44.199758574Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdDatabaseQuotaLowSpace\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \\u003e 95\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md\",\"summary\":\"etcd cluster database is running full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000176359,\"lastEvaluation\":\"2024-04-12T22:36:44.199396135Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdExcessiveDatabaseGrowth\",\"query\":\"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \\u003e etcd_server_quota_backend_bytes\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.\",\"summary\":\"etcd cluster database growing very fast.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.0001838,\"lastEvaluation\":\"2024-04-12T22:36:44.199573524Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighCommitDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.25\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile commit durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00036941,\"lastEvaluation\":\"2024-04-12T22:36:44.199025365Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000312917,\"lastEvaluation\":\"2024-04-12T22:36:44.198711528Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.5\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00038715,\"lastEvaluation\":\"2024-04-12T22:36:44.198323418Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedProposals\",\"query\":\"rate(etcd_server_proposals_failed_total{job=~\\\".*etcd.*\\\"}[15m]) \\u003e 5\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of proposal failures.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000140508,\"lastEvaluation\":\"2024-04-12T22:36:44.19818184Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMemberCommunicationSlow\",\"query\":\"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.15\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster member communication is slow.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000718188,\"lastEvaluation\":\"2024-04-12T22:36:44.197462382Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMembersDown\",\"query\":\"max without (endpoint) (sum without (instance) (up{job=~\\\".*etcd.*\\\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\\\".*etcd.*\\\"}[2m])) \\u003e 0.01)) \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": members are down ({{ $value }}).\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md\",\"summary\":\"etcd cluster members are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000461165,\"lastEvaluation\":\"2024-04-12T22:36:44.196853459Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdNoLeader\",\"query\":\"etcd_server_has_leader{job=~\\\".*etcd.*\\\"} == 0\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member {{ $labels.instance }} has no leader.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md\",\"summary\":\"etcd cluster has no leader.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000144798,\"lastEvaluation\":\"2024-04-12T22:36:44.197316204Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00318783,\"lastEvaluation\":\"2024-04-12T22:36:44.196846789Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-bdafb11d-bf7f-4480-ae4e-7cbeddeea773.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdGRPCRequestsSlow\",\"query\":\"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\\\"Defragment\\\",grpc_type=\\\"unary\\\",job=\\\"etcd\\\"}[10m]))) \\u003e 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md\",\"summary\":\"etcd grpc requests are slow\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.009185464,\"lastEvaluation\":\"2024-04-12T22:36:24.417977275Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 50\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.012922265,\"lastEvaluation\":\"2024-04-12T22:36:24.444522378Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 10\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.017354379,\"lastEvaluation\":\"2024-04-12T22:36:24.427166069Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfLeaderChanges\",\"query\":\"avg(changes(etcd_server_is_leader[10m])) \\u003e 5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.\",\"summary\":\"etcd cluster has high number of leader changes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000164563,\"lastEvaluation\":\"2024-04-12T22:36:24.457446793Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdInsufficientMembers\",\"query\":\"sum without (instance, pod) (up{job=\\\"etcd\\\"} == bool 1 and etcd_server_has_leader{job=\\\"etcd\\\"} == bool 1) \\u003c ((count without (instance, pod) (up{job=\\\"etcd\\\"}) + 1) / 2)\",\"duration\":180,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md\",\"summary\":\"etcd is reporting that a majority of instances are unavailable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000318496,\"lastEvaluation\":\"2024-04-12T22:36:24.457612596Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.037828601,\"lastEvaluation\":\"2024-04-12T22:36:24.417657431Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imageregistry.operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-2e046cbc-5e3f-419b-9e59-085c6eb3d543.yaml\",\"rules\":[{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.Create\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000205973,\"lastEvaluation\":\"2024-04-12T22:36:27.142541754Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.ServeBlob\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000348556,\"lastEvaluation\":\"2024-04-12T22:36:27.142191437Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Get\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000202683,\"lastEvaluation\":\"2024-04-12T22:36:27.142749038Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Put\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000142263,\"lastEvaluation\":\"2024-04-12T22:36:27.142952901Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001056526,\"lastEvaluation\":\"2024-04-12T22:36:27.141155686Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imagestreams.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-62089bf6-dfc3-4319-af63-38d6191b31ea.yaml\",\"rules\":[{\"name\":\"imageregistry:imagestreamtags_count:sum\",\"query\":\"sum by (location, source) (image_registry_image_stream_tags_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000297936,\"lastEvaluation\":\"2024-04-12T22:36:29.517918454Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000318487,\"lastEvaluation\":\"2024-04-12T22:36:29.517397281Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress-to-route-controller.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-dfd4b459-b669-489c-9020-22defa539ee2.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"IngressWithoutClassName\",\"query\":\"openshift_ingress_to_route_controller_ingress_without_class_name == 1\",\"duration\":86400,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"alerts\":[{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.130.0.46:8443\",\"job\":\"route-controller-manager\",\"name\":\"test\",\"namespace\":\"e2e-test-router-stress-g64j9\",\"pod\":\"route-controller-manager-5bdb9f48f4-nbc4h\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress test is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:21:56.656877779Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.00057098,\"lastEvaluation\":\"2024-04-12T22:36:26.658168642Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UnmanagedRoutes\",\"query\":\"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is a Route owned by an unmanaged Ingress.\",\"message\":\"Route {{ $labels.name }} is owned by an unmanaged Ingress.\",\"summary\":\"Route owned by an Ingress no longer managed\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00055756,\"lastEvaluation\":\"2024-04-12T22:36:26.658742192Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00108014,\"lastEvaluation\":\"2024-04-12T22:36:26.657113115Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-dfd4b459-b669-489c-9020-22defa539ee2.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HAProxyDown\",\"query\":\"haproxy_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"This alert fires when metrics report that HAProxy is down.\",\"message\":\"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}\",\"summary\":\"HAProxy is down\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000117152,\"lastEvaluation\":\"2024-04-12T22:36:36.029997778Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HAProxyReloadFail\",\"query\":\"template_router_reload_failure == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.\",\"message\":\"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes\",\"summary\":\"HAProxy reload failure\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000192924,\"lastEvaluation\":\"2024-04-12T22:36:36.029803404Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerDegraded\",\"query\":\"ingress_controller_conditions{condition=\\\"Degraded\\\"} == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController status is degraded.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\ndegraded: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is degraded\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000116962,\"lastEvaluation\":\"2024-04-12T22:36:36.0301163Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerUnavailable\",\"query\":\"ingress_controller_conditions{condition=\\\"Available\\\"} == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController is not available.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\nunavailable: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is unavailable\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000086392,\"lastEvaluation\":\"2024-04-12T22:36:36.030234183Z\",\"type\":\"alerting\"},{\"name\":\"cluster:openshift_route_info:tls_termination:sum\",\"query\":\"sum by (tls_termination) (openshift_route_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000177153,\"lastEvaluation\":\"2024-04-12T22:36:36.030713762Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:avg\",\"query\":\"avg(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000077671,\"lastEvaluation\":\"2024-04-12T22:36:36.030548629Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:max\",\"query\":\"max(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000088982,\"lastEvaluation\":\"2024-04-12T22:36:36.030458617Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:median\",\"query\":\"quantile(0.5, route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000085012,\"lastEvaluation\":\"2024-04-12T22:36:36.0306273Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:min\",\"query\":\"min(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000134313,\"lastEvaluation\":\"2024-04-12T22:36:36.030321674Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001099001,\"lastEvaluation\":\"2024-04-12T22:36:36.029794314Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"insights\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-9ee17981-1562-4044-be51-c623d1ec6f10.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"InsightsDisabled\",\"query\":\"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\\\"Disabled\\\",name=\\\"insights\\\"} == 1)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html\",\"summary\":\"Insights operator is disabled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000255843,\"lastEvaluation\":\"2024-04-12T22:36:26.980475055Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"InsightsRecommendationActive\",\"query\":\"insights_recommendation_active == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights recommendation \\\"{{ $labels.description }}\\\" with total risk \\\"{{ $labels.total_risk }}\\\" was detected on the cluster. More information is available at {{ $labels.info_link }}.\",\"summary\":\"An Insights recommendation is active for this cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077884,\"lastEvaluation\":\"2024-04-12T22:36:26.980878496Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SimpleContentAccessNotAvailable\",\"query\":\"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\\\"SCAAvailable\\\",name=\\\"insights\\\",reason=\\\"NotFound\\\"}[5m]) == 0)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.\",\"summary\":\"Simple content access certificates are not available.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000144768,\"lastEvaluation\":\"2024-04-12T22:36:26.980732438Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000520169,\"lastEvaluation\":\"2024-04-12T22:36:26.980225537Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pre-release-lifecycle\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-0d5178dd-4e23-49ca-884b-5d9c83f14fb4.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"APIRemovedInNextEUSReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.29\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next EUS version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.059891486,\"lastEvaluation\":\"2024-04-12T22:36:49.753094018Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"APIRemovedInNextReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.29\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.057387063,\"lastEvaluation\":\"2024-04-12T22:36:49.695703445Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.058073556,\"lastEvaluation\":\"2024-04-12T22:36:49.680137292Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-audit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-3528d707-66c1-425c-bbdb-459ae3d79e76.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AuditLogError\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\\\".+-apiserver\\\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\\\".+-apiserver\\\"}[5m])) \\u003e 0\",\"duration\":60,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"An API Server had an error writing to an audit log.\",\"summary\":\"An API Server instance was unable to write audit logs. This could be\\ntriggered by the node running out of space, or a malicious actor\\ntampering with the audit logs.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000601002,\"lastEvaluation\":\"2024-04-12T22:36:35.38840949Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000720889,\"lastEvaluation\":\"2024-04-12T22:36:35.388292604Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"control-plane-cpu-utilization\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-7d5403fb-3bd5-4f24-ada6-427ad465fbde.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000343136,\"lastEvaluation\":\"2024-04-12T22:36:36.635727642Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000291696,\"lastEvaluation\":\"2024-04-12T22:36:36.635435026Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneCPU\",\"query\":\"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / count(kube_node_role{role=\\\"master\\\"}) \\u003e 60\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000682083,\"lastEvaluation\":\"2024-04-12T22:36:36.634751593Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001586624,\"lastEvaluation\":\"2024-04-12T22:36:36.634190473Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"api-performance\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-60c107f0-5c52-40c2-b772-4473cfac1e20.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000199404,\"lastEvaluation\":\"2024-04-12T22:36:23.675383044Z\",\"type\":\"recording\"},{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000169833,\"lastEvaluation\":\"2024-04-12T22:36:23.675583668Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.023276721,\"lastEvaluation\":\"2024-04-12T22:36:23.481059232Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.026017973,\"lastEvaluation\":\"2024-04-12T22:36:23.504341813Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:1m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000294435,\"lastEvaluation\":\"2024-04-12T22:36:53.434552368Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:5m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000239162,\"lastEvaluation\":\"2024-04-12T22:36:53.434848284Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003580368,\"lastEvaluation\":\"2024-04-12T22:36:23.654924816Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003470766,\"lastEvaluation\":\"2024-04-12T22:36:23.658508734Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00057902,\"lastEvaluation\":\"2024-04-12T22:36:23.647286082Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000487939,\"lastEvaluation\":\"2024-04-12T22:36:23.64826652Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001961498,\"lastEvaluation\":\"2024-04-12T22:36:23.643433938Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001887095,\"lastEvaluation\":\"2024-04-12T22:36:23.645397016Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004412413,\"lastEvaluation\":\"2024-04-12T22:36:23.614417119Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004283691,\"lastEvaluation\":\"2024-04-12T22:36:23.618832142Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009910747,\"lastEvaluation\":\"2024-04-12T22:36:23.623383119Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010133423,\"lastEvaluation\":\"2024-04-12T22:36:23.633297177Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000130022,\"lastEvaluation\":\"2024-04-12T22:36:23.623118444Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000131973,\"lastEvaluation\":\"2024-04-12T22:36:23.623249546Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002740862,\"lastEvaluation\":\"2024-04-12T22:36:23.609015826Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002656361,\"lastEvaluation\":\"2024-04-12T22:36:23.611758628Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002852824,\"lastEvaluation\":\"2024-04-12T22:36:23.667102767Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002514648,\"lastEvaluation\":\"2024-04-12T22:36:23.669957711Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002680531,\"lastEvaluation\":\"2024-04-12T22:36:23.66198185Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002435906,\"lastEvaluation\":\"2024-04-12T22:36:23.664664841Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002906495,\"lastEvaluation\":\"2024-04-12T22:36:23.672474479Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000214661,\"lastEvaluation\":\"2024-04-12T22:36:53.434146596Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00018618,\"lastEvaluation\":\"2024-04-12T22:36:53.434363498Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00211391,\"lastEvaluation\":\"2024-04-12T22:36:23.576715704Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.022041597,\"lastEvaluation\":\"2024-04-12T22:36:23.530366036Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.021222472,\"lastEvaluation\":\"2024-04-12T22:36:23.552412404Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007422281,\"lastEvaluation\":\"2024-04-12T22:36:23.57913066Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007539133,\"lastEvaluation\":\"2024-04-12T22:36:23.586555421Z\",\"type\":\"recording\"},{\"name\":\"instance_request_kind:apiserver_current_inflight_requests:sum\",\"query\":\"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000296876,\"lastEvaluation\":\"2024-04-12T22:36:23.578832524Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.091631305,\"lastEvaluation\":\"2024-04-12T22:36:51.904027503Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.102604299,\"lastEvaluation\":\"2024-04-12T22:36:51.996960149Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.32751045,\"lastEvaluation\":\"2024-04-12T22:36:52.725709768Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.336176817,\"lastEvaluation\":\"2024-04-12T22:36:53.012343403Z\",\"type\":\"recording\"},{\"name\":\"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum\",\"query\":\"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000398517,\"lastEvaluation\":\"2024-04-12T22:36:23.647866783Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009125995,\"lastEvaluation\":\"2024-04-12T22:36:53.399889504Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008553235,\"lastEvaluation\":\"2024-04-12T22:36:53.409018859Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003067948,\"lastEvaluation\":\"2024-04-12T22:36:23.573644906Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_storage_objects:max\",\"query\":\"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002858074,\"lastEvaluation\":\"2024-04-12T22:36:23.675755251Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000373647,\"lastEvaluation\":\"2024-04-12T22:36:23.654257964Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000288945,\"lastEvaluation\":\"2024-04-12T22:36:23.654634331Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.283265351,\"lastEvaluation\":\"2024-04-12T22:36:51.386615443Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.234128367,\"lastEvaluation\":\"2024-04-12T22:36:51.669889875Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.02574044,\"lastEvaluation\":\"2024-04-12T22:36:53.34852389Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.025612693,\"lastEvaluation\":\"2024-04-12T22:36:53.37426992Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007143945,\"lastEvaluation\":\"2024-04-12T22:36:23.594097204Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007768227,\"lastEvaluation\":\"2024-04-12T22:36:23.601244029Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002966196,\"lastEvaluation\":\"2024-04-12T22:36:23.648756859Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002529138,\"lastEvaluation\":\"2024-04-12T22:36:23.651726346Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.202137845,\"lastEvaluation\":\"2024-04-12T22:36:52.287428993Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.213206304,\"lastEvaluation\":\"2024-04-12T22:36:52.512496584Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.090115927,\"lastEvaluation\":\"2024-04-12T22:36:52.099571859Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.093912566,\"lastEvaluation\":\"2024-04-12T22:36:52.193510476Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007978605,\"lastEvaluation\":\"2024-04-12T22:36:53.417574974Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008586077,\"lastEvaluation\":\"2024-04-12T22:36:53.425556339Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":2.215395111,\"lastEvaluation\":\"2024-04-12T22:36:21.386081653Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-requests-in-flight\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-6824c540-c854-48f2-9d56-c5eabf4a2b2f.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m\",\"query\":\"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver\\\"})[2m:])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000474705,\"lastEvaluation\":\"2024-04-12T22:36:27.870854345Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000492436,\"lastEvaluation\":\"2024-04-12T22:36:27.870839924Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-basic\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4f08eea7-5a77-4739-ba5f-4ebfa6a5c384.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1h) \\u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \\u003e (14.4 * 0.01)\",\"duration\":120,\"labels\":{\"long\":\"1h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"5m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000372347,\"lastEvaluation\":\"2024-04-12T22:36:25.801155645Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate6h) \\u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \\u003e (6 * 0.01)\",\"duration\":900,\"labels\":{\"long\":\"6h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"30m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000186373,\"lastEvaluation\":\"2024-04-12T22:36:25.801529542Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000451685,\"lastEvaluation\":\"2024-04-12T22:36:25.801132033Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4f08eea7-5a77-4739-ba5f-4ebfa6a5c384.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.015164194,\"lastEvaluation\":\"2024-04-12T22:36:38.558555321Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.010181167,\"lastEvaluation\":\"2024-04-12T22:36:38.588315118Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.009728989,\"lastEvaluation\":\"2024-04-12T22:36:38.548823922Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.007657447,\"lastEvaluation\":\"2024-04-12T22:36:38.598549263Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.010208408,\"lastEvaluation\":\"2024-04-12T22:36:38.538611644Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.005851462,\"lastEvaluation\":\"2024-04-12T22:36:38.606210031Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.014587073,\"lastEvaluation\":\"2024-04-12T22:36:38.573724735Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.009160747,\"lastEvaluation\":\"2024-04-12T22:36:38.612063442Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.121687879,\"lastEvaluation\":\"2024-04-12T22:36:39.048674253Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.130384218,\"lastEvaluation\":\"2024-04-12T22:36:38.918282635Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.123482134,\"lastEvaluation\":\"2024-04-12T22:36:38.794796241Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.083649567,\"lastEvaluation\":\"2024-04-12T22:36:38.637102042Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.073791771,\"lastEvaluation\":\"2024-04-12T22:36:38.72098503Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.008557175,\"lastEvaluation\":\"2024-04-12T22:36:38.621226979Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.007312438,\"lastEvaluation\":\"2024-04-12T22:36:38.629786694Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.631766988,\"lastEvaluation\":\"2024-04-12T22:36:38.538600094Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-extended\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-168fb4ca-48ed-4fa6-95a7-20c6c46562ef.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1d) \\u003e (3 * 0.01) and sum(apiserver_request:burnrate2h) \\u003e (3 * 0.01)\",\"duration\":3600,\"labels\":{\"long\":\"1d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"2h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000321587,\"lastEvaluation\":\"2024-04-12T22:36:35.100389895Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate3d) \\u003e (1 * 0.01) and sum(apiserver_request:burnrate6h) \\u003e (1 * 0.01)\",\"duration\":10800,\"labels\":{\"long\":\"3d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"6h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000424182,\"lastEvaluation\":\"2024-04-12T22:36:35.10077552Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001123449,\"lastEvaluation\":\"2024-04-12T22:36:35.100083524Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-168fb4ca-48ed-4fa6-95a7-20c6c46562ef.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.019031292,\"lastEvaluation\":\"2024-04-12T22:36:31.518244181Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.010686038,\"lastEvaluation\":\"2024-04-12T22:36:31.553264243Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.022732989,\"lastEvaluation\":\"2024-04-12T22:36:31.495505362Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.012240721,\"lastEvaluation\":\"2024-04-12T22:36:31.563953511Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.015979289,\"lastEvaluation\":\"2024-04-12T22:36:31.537281113Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.011007505,\"lastEvaluation\":\"2024-04-12T22:36:31.576198032Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.091709845,\"lastEvaluation\":\"2024-04-12T22:36:31.495498572Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-5005c833-ea1b-4666-ae08-3bea362dc2d3.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TechPreviewNoUpgrade\",\"query\":\"cluster_feature_set{name!=\\\"\\\",namespace=\\\"openshift-kube-apiserver-operator\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.\",\"summary\":\"Cluster has enabled tech preview features that will prevent upgrades.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002395806,\"lastEvaluation\":\"2024-04-12T22:36:45.48300113Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002119112,\"lastEvaluation\":\"2024-04-12T22:36:45.482834302Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pod-security-violation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-be42b924-fa88-4cdc-b097-1e5e3015f2ef.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace!=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created in namespace \\\"{{ $labels.ocp_namespace }}\\\" but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000136227,\"lastEvaluation\":\"2024-04-12T22:36:27.163648332Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads users created in the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000253205,\"lastEvaluation\":\"2024-04-12T22:36:27.163347548Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000380587,\"lastEvaluation\":\"2024-04-12T22:36:27.163343618Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-59a05266-d32f-408a-b2c9-134a0aa47a1f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"GarbageCollectorSyncFailed\",\"query\":\"rate(garbagecollector_controller_resources_sync_error_total[5m]) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md\",\"summary\":\"There was a problem with syncing the resources for garbage collection.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000191034,\"lastEvaluation\":\"2024-04-12T22:36:26.687631783Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeControllerManagerDown\",\"query\":\"absent(up{job=\\\"kube-controller-manager\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-controller-manager\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeControllerManager has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000311836,\"lastEvaluation\":\"2024-04-12T22:36:26.686338099Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetAtLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \\u003e 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md\",\"summary\":\"The pod disruption budget is preventing further disruption to pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000466764,\"lastEvaluation\":\"2024-04-12T22:36:26.686652636Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \\u003c kube_poddisruptionbudget_status_desired_healthy)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md\",\"summary\":\"The pod disruption budget registers insufficient amount of pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000365197,\"lastEvaluation\":\"2024-04-12T22:36:26.687264646Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00158003,\"lastEvaluation\":\"2024-04-12T22:36:26.686246467Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-14fb49f3-f1fb-4874-93f5-bc12c6437bcf.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeSchedulerDown\",\"query\":\"absent(up{job=\\\"scheduler\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-scheduler\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeScheduler has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000321926,\"lastEvaluation\":\"2024-04-12T22:36:31.339316817Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000336017,\"lastEvaluation\":\"2024-04-12T22:36:31.339306417Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"scheduler-legacy-policy-deprecated\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-14fb49f3-f1fb-4874-93f5-bc12c6437bcf.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SchedulerLegacyPolicySet\",\"query\":\"cluster_legacy_scheduler_policy \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.\",\"summary\":\"Legacy scheduler policy API in use by the scheduler.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000216224,\"lastEvaluation\":\"2024-04-12T22:36:43.443324055Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000229705,\"lastEvaluation\":\"2024-04-12T22:36:43.443314645Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-api-operator-metrics-collector-up\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineAPIOperatorMetricsCollectionFailing\",\"query\":\"mapi_mao_collector_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"For more details:  oc logs \\u003cmachine-api-operator-pod-name\\u003e -n openshift-machine-api\",\"summary\":\"machine api operator metrics collection is failing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000214694,\"lastEvaluation\":\"2024-04-12T22:36:23.972886041Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000226154,\"lastEvaluation\":\"2024-04-12T22:36:23.972879051Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-health-check-unterminated-short-circuit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineHealthCheckUnterminatedShortCircuit\",\"query\":\"mapi_machinehealthcheck_short_circuit == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\\nthe status of machines in the cluster.\\n\",\"summary\":\"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000221112,\"lastEvaluation\":\"2024-04-12T22:36:51.928398519Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000233643,\"lastEvaluation\":\"2024-04-12T22:36:51.928389159Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-not-yet-deleted\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineNotYetDeleted\",\"query\":\"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\\\"Deleting\\\"}[15m])) \\u003e 0\",\"duration\":21600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine is not properly deleting, this may be due to a configuration issue with the\\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\\nlong termination periods which are preventing deletion.\\n\",\"summary\":\"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000272524,\"lastEvaluation\":\"2024-04-12T22:36:44.941617306Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000285856,\"lastEvaluation\":\"2024-04-12T22:36:44.941608115Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-with-no-running-phase\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithNoRunningPhase\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\\\"Running|Deleting\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine has been without a Running or Deleting phase for more than 60 minutes.\\nThe machine may not have been provisioned properly from the infrastructure provider, or\\nit might have issues with CertificateSigningRequests being approved.\\n\",\"summary\":\"machine {{ $labels.name }} is in phase: {{ $labels.phase }}\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000326337,\"lastEvaluation\":\"2024-04-12T22:36:31.397113273Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000264484,\"lastEvaluation\":\"2024-04-12T22:36:31.396641077Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-without-valid-node-ref\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithoutValidNode\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"If the machine never became a node, you should diagnose the machine related failures.\\nIf the node was deleted from the API, you may delete the machine if appropriate.\\n\",\"summary\":\"machine {{ $labels.name }} does not have valid node reference\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000335148,\"lastEvaluation\":\"2024-04-12T22:36:45.843108201Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000363587,\"lastEvaluation\":\"2024-04-12T22:36:45.836234369Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-drain-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCDrainError\",\"query\":\"mcc_drain_err \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000240684,\"lastEvaluation\":\"2024-04-12T22:36:26.126555741Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000254695,\"lastEvaluation\":\"2024-04-12T22:36:26.126545581Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-pool-alert\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCPoolAlert\",\"query\":\"mcc_pool_alert \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000245005,\"lastEvaluation\":\"2024-04-12T22:36:45.350111834Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000257436,\"lastEvaluation\":\"2024-04-12T22:36:45.350103014Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"os-image-override.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"name\":\"os_image_url_override:sum\",\"query\":\"sum(os_image_url_override)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000269855,\"lastEvaluation\":\"2024-04-12T22:36:25.675348321Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000312607,\"lastEvaluation\":\"2024-04-12T22:36:25.675173108Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"extremely-high-individual-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneMemory\",\"query\":\"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 90\",\"duration\":2700,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.\",\"summary\":\"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000650663,\"lastEvaluation\":\"2024-04-12T22:36:39.133743062Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000675846,\"lastEvaluation\":\"2024-04-12T22:36:39.132725259Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"high-overall-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneMemory\",\"query\":\"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 60\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.\",\"summary\":\"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000830794,\"lastEvaluation\":\"2024-04-12T22:36:37.080318668Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000622492,\"lastEvaluation\":\"2024-04-12T22:36:37.079547936Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-kubelet-health-state-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeletHealthState\",\"query\":\"mcd_kubelet_state \\u003e 2\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet health failure threshold reached\",\"summary\":\"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000346499,\"lastEvaluation\":\"2024-04-12T22:36:36.713249675Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000241255,\"lastEvaluation\":\"2024-04-12T22:36:36.712959868Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-pivot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDPivotError\",\"query\":\"mcd_pivot_errors_total \\u003e 0\",\"duration\":120,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000243424,\"lastEvaluation\":\"2024-04-12T22:36:26.054822042Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000256836,\"lastEvaluation\":\"2024-04-12T22:36:26.054811371Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-reboot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDRebootError\",\"query\":\"mcd_reboots_failed_total \\u003e 0\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000279045,\"lastEvaluation\":\"2024-04-12T22:36:24.37248846Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000271625,\"lastEvaluation\":\"2024-04-12T22:36:24.3717978Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"system-memory-exceeds-reservation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SystemMemoryExceedsReservation\",\"query\":\"sum by (node) (container_memory_rss{id=\\\"/system.slice\\\"}) \\u003e ((sum by (node) (kube_node_status_capacity{resource=\\\"memory\\\"} - kube_node_status_allocatable{resource=\\\"memory\\\"})) * 0.95)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).\",\"summary\":\"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000574741,\"lastEvaluation\":\"2024-04-12T22:36:44.345946605Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000548709,\"lastEvaluation\":\"2024-04-12T22:36:44.345632125Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"operator.marketplace.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-marketplace-marketplace-alert-rules-714df196-fc7d-484a-8308-076cc71239c1.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"OperatorHubSourceError\",\"query\":\"catalogsource_ready{exported_namespace=\\\"openshift-marketplace\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Operators shipped via the {{ $labels.name }} source are not available for installation until the issue is fixed. Operators already installed from this source will not receive updates until issue is fixed. Inspect the status of the pod owned by {{ $labels.name }} source in the openshift-marketplace namespace (oc -n openshift-marketplace get pods -l olm.catalogSource={{ $labels.name }}) to diagnose and repair.\",\"summary\":\"The {{ $labels.name }} source is in non-ready state for more than 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000205474,\"lastEvaluation\":\"2024-04-12T22:36:26.137690192Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000213214,\"lastEvaluation\":\"2024-04-12T22:36:26.137684882Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"alertmanager.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-alertmanager-main-rules-b89ef7fc-8e25-4a61-90d5-f3ea1f8d7983.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterDown\",\"query\":\"(count by (namespace, service) (avg_over_time(up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c 0.5) / count by (namespace, service) (up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) \\u003e= 0.5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of Alertmanager instances within the {{$labels.job}} cluster have been up for less than half of the last 5m.\",\"summary\":\"Half or more of the Alertmanager instances within the same cluster are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000275936,\"lastEvaluation\":\"2024-04-12T22:36:40.32919809Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterFailedToSendAlerts\",\"query\":\"min by (namespace, service, integration) (rate(alertmanager_notifications_failed_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The minimum notification failure rate to {{ $labels.integration }} sent from any instance in the {{$labels.job}} cluster is {{ $value | humanizePercentage }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerClusterFailedToSendAlerts.md\",\"summary\":\"All Alertmanager instances in a cluster failed to send notifications to a critical integration.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000824656,\"lastEvaluation\":\"2024-04-12T22:36:40.328170879Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerConfigInconsistent\",\"query\":\"count by (namespace, service) (count_values by (namespace, service) (\\\"config_hash\\\", alertmanager_config_hash{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) != 1\",\"duration\":1200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager instances within the {{$labels.job}} cluster have different configurations.\",\"summary\":\"Alertmanager instances within the same cluster have different configurations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000199453,\"lastEvaluation\":\"2024-04-12T22:36:40.328996846Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedReload\",\"query\":\"max_over_time(alertmanager_config_last_reload_successful{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Configuration has failed to load for {{ $labels.namespace }}/{{ $labels.pod}}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedReload.md\",\"summary\":\"Reloading an Alertmanager configuration has failed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000355179,\"lastEvaluation\":\"2024-04-12T22:36:40.326475027Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedToSendAlerts\",\"query\":\"(rate(alertmanager_notifications_failed_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} failed to send {{ $value | humanizePercentage }} of notifications to {{ $labels.integration }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedToSendAlerts.md\",\"summary\":\"An Alertmanager instance failed to send notifications.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000700037,\"lastEvaluation\":\"2024-04-12T22:36:40.327184894Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerMembersInconsistent\",\"query\":\"max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c on (namespace, service) group_left () count by (namespace, service) (max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]))\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} has only found {{ $value }} members of the {{$labels.job}} cluster.\",\"summary\":\"A member of an Alertmanager cluster has not found all other cluster members.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000351658,\"lastEvaluation\":\"2024-04-12T22:36:40.326831955Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003114101,\"lastEvaluation\":\"2024-04-12T22:36:40.326362195Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-list-watch.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"apiserver_list_watch_request_success_total:rate:sum\",\"query\":\"sum by (verb) (rate(apiserver_request_total{code=~\\\"2..\\\",verb=~\\\"LIST|WATCH\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011091991,\"lastEvaluation\":\"2024-04-12T22:36:49.03612487Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.01343978,\"lastEvaluation\":\"2024-04-12T22:36:49.035981344Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"Watchdog\",\"query\":\"vector(1)\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"alerts\":[{\"labels\":{\"alertname\":\"Watchdog\",\"namespace\":\"openshift-monitoring\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"state\":\"firing\",\"activeAt\":\"2024-04-12T21:38:00.04805628Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.00036572,\"lastEvaluation\":\"2024-04-12T22:36:30.049373349Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000325186,\"lastEvaluation\":\"2024-04-12T22:36:30.049005388Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"count:up0\",\"query\":\"count without (instance, pod, node) (up == 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000584401,\"lastEvaluation\":\"2024-04-12T22:36:30.562035441Z\",\"type\":\"recording\"},{\"name\":\"count:up1\",\"query\":\"count without (instance, pod, node) (up == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001188013,\"lastEvaluation\":\"2024-04-12T22:36:30.560845548Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001786285,\"lastEvaluation\":\"2024-04-12T22:36:30.560835887Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-node-recording.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:node_cpu:ratio\",\"query\":\"cluster:node_cpu:sum_rate5m / count(sum by (instance, cpu) (node_cpu_seconds_total))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001001909,\"lastEvaluation\":\"2024-04-12T22:36:31.717688757Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:sum_rate5m\",\"query\":\"sum(rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000501319,\"lastEvaluation\":\"2024-04-12T22:36:31.717186508Z\",\"type\":\"recording\"},{\"name\":\"instance:node_cpu:rate:sum\",\"query\":\"sum by (instance) (rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000744524,\"lastEvaluation\":\"2024-04-12T22:36:31.715888793Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_receive_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000298946,\"lastEvaluation\":\"2024-04-12T22:36:31.716635227Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_transmit_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_transmit_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000250275,\"lastEvaluation\":\"2024-04-12T22:36:31.716935283Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002811523,\"lastEvaluation\":\"2024-04-12T22:36:31.715881853Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-recurring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:usage:workload:capacity_physical_cpu_core_seconds\",\"query\":\"sum_over_time(workload:capacity_physical_cpu_cores:sum[30s:1s]) + ((cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) or (absent(cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) * 0))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000427389,\"lastEvaluation\":\"2024-04-12T22:36:34.377895604Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000439449,\"lastEvaluation\":\"2024-04-12T22:36:34.377886824Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-network\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeNetworkInterfaceFlapping\",\"query\":\"changes(node_network_up{device!~\\\"veth.+|tunbr\\\",job=\\\"node-exporter\\\"}[2m]) \\u003e 2\",\"duration\":120,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Network interface \\\"{{ $labels.device }}\\\" changing its up status often on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\",\"summary\":\"Network interface is often changing its status\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000479155,\"lastEvaluation\":\"2024-04-12T22:36:25.834301237Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000651173,\"lastEvaluation\":\"2024-04-12T22:36:25.833252363Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-build.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"openshift:build_by_strategy:sum\",\"query\":\"sum by (strategy) (openshift_build_status_phase_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000352997,\"lastEvaluation\":\"2024-04-12T22:36:39.546313771Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000366027,\"lastEvaluation\":\"2024-04-12T22:36:39.546304101Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd-telemetry.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000471499,\"lastEvaluation\":\"2024-04-12T22:36:46.87614077Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000474385,\"lastEvaluation\":\"2024-04-12T22:36:46.874714269Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000240762,\"lastEvaluation\":\"2024-04-12T22:36:46.874471787Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_use_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000215515,\"lastEvaluation\":\"2024-04-12T22:36:46.875923125Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000687606,\"lastEvaluation\":\"2024-04-12T22:36:46.875190305Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002345756,\"lastEvaluation\":\"2024-04-12T22:36:46.874268973Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TargetDown\",\"query\":\"100 * ((1 - sum by (job, namespace, service) (up and on (namespace, pod) kube_pod_info) / count by (job, namespace, service) (up and on (namespace, pod) kube_pod_info)) or (count by (job, namespace, service) (up == 0) / count by (job, namespace, service) (up))) \\u003e 10\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ printf \\\"%.4g\\\" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.\",\"summary\":\"Some targets were not reachable from the monitoring server for an extended period of time.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00725452,\"lastEvaluation\":\"2024-04-12T22:36:49.131392564Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00726747,\"lastEvaluation\":\"2024-04-12T22:36:49.131383473Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:ingress_controller_aws_nlb_active:sum\",\"query\":\"sum(ingress_controller_aws_nlb_active) or vector(0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000117253,\"lastEvaluation\":\"2024-04-12T22:36:47.632569255Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_in:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_in_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000150232,\"lastEvaluation\":\"2024-04-12T22:36:47.628183781Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_out:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_out_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000159203,\"lastEvaluation\":\"2024-04-12T22:36:47.628335493Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_connections:sum\",\"query\":\"sum(haproxy_frontend_current_sessions)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000108811,\"lastEvaluation\":\"2024-04-12T22:36:47.628495827Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace=~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00157298,\"lastEvaluation\":\"2024-04-12T22:36:47.629900964Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001092682,\"lastEvaluation\":\"2024-04-12T22:36:47.631475284Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace!~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000785795,\"lastEvaluation\":\"2024-04-12T22:36:47.628605789Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00050599,\"lastEvaluation\":\"2024-04-12T22:36:47.629393424Z\",\"type\":\"recording\"},{\"name\":\"code:cluster:ingress_http_request_count:rate5m:sum\",\"query\":\"sum by (code) (rate(haproxy_server_http_responses_total[5m]) \\u003e 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001585241,\"lastEvaluation\":\"2024-04-12T22:36:47.62659673Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006102148,\"lastEvaluation\":\"2024-04-12T22:36:47.62658617Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-kubernetes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"AlertmanagerReceiversNotConfigured\",\"query\":\"cluster:alertmanager_integrations:max == 0\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"alerts\":[{\"labels\":{\"alertname\":\"AlertmanagerReceiversNotConfigured\",\"namespace\":\"openshift-monitoring\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"state\":\"firing\",\"activeAt\":\"2024-04-12T21:39:17.894182481Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000165894,\"lastEvaluation\":\"2024-04-12T22:36:47.996157584Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterMonitoringOperatorReconciliationErrors\",\"query\":\"max_over_time(cluster_monitoring_operator_last_reconciliation_successful[5m]) == 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors are occurring during reconciliation cycles. Inspect the cluster-monitoring-operator log for potential root causes.\",\"summary\":\"Cluster Monitoring Operator is experiencing unexpected reconciliation errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077241,\"lastEvaluation\":\"2024-04-12T22:36:47.996079603Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentReplicasMismatch\",\"query\":\"(((kube_deployment_spec_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_deployment_status_replicas_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_deployment_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)) * on () group_left () cluster:control_plane:all_nodes_ready) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes. This indicates that cluster infrastructure is unable to start or restart the necessary components. This most often occurs when one or more nodes are down or partioned from the cluster, or a fault occurs on the node that prevents the workload from starting. In rare cases this may indicate a new version of a cluster component cannot start due to a bug or configuration error. Assess the pods for this deployment to verify they are running on healthy nodes and then contact support.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeDeploymentReplicasMismatch.md\",\"summary\":\"Deployment has not matched the expected number of replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002321535,\"lastEvaluation\":\"2024-04-12T22:36:47.996324498Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodNotScheduled\",\"query\":\"last_over_time(kube_pod_status_unschedulable{namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} cannot be scheduled for more than 30 minutes.\\nCheck the details of the pod with the following command:\\noc describe -n {{ $labels.namespace }} pod {{ $labels.pod }}\",\"summary\":\"Pod cannot be scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000541001,\"lastEvaluation\":\"2024-04-12T22:36:48.025027584Z\",\"type\":\"alerting\"},{\"name\":\"cluster:alertmanager_integrations:max\",\"query\":\"max(alertmanager_integrations{namespace=\\\"openshift-monitoring\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000121752,\"lastEvaluation\":\"2024-04-12T22:36:47.995154745Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (label_replace(cluster:infra_nodes, \\\"label_node_role_kubernetes_io\\\", \\\"infra\\\", \\\"\\\", \\\"\\\") * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000464569,\"lastEvaluation\":\"2024-04-12T22:36:47.964889868Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000130143,\"lastEvaluation\":\"2024-04-12T22:36:47.966423848Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_sockets_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (max by (node, package, label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000144733,\"lastEvaluation\":\"2024-04-12T22:36:47.994840189Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_memory_bytes:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000365347,\"lastEvaluation\":\"2024-04-12T22:36:47.966555301Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_cpu_usage:ratio\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m])) / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008544355,\"lastEvaluation\":\"2024-04-12T22:36:47.952716206Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_spec_cpu_shares:ratio\",\"query\":\"sum(container_spec_cpu_shares{container=\\\"\\\",pod!=\\\"\\\"}) / 1000 / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008869612,\"lastEvaluation\":\"2024-04-12T22:36:47.943845054Z\",\"type\":\"recording\"},{\"name\":\"cluster:control_plane:all_nodes_ready\",\"query\":\"sum(min by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}) and max by (node) (kube_node_role{role=\\\"master\\\"})) == bool sum(kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000271916,\"lastEvaluation\":\"2024-04-12T22:36:47.995738736Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_hyperthreading\",\"query\":\"clamp_max(label_replace(sum by (instance, package, core) (node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_replace(label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003e 1, \\\"label_node_hyperthread_enabled\\\", \\\"true\\\", \\\"instance\\\", \\\"(.*)\\\") or on (instance, package) label_replace(sum by (instance, package, core) (label_replace(node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003c= 1, \\\"label_node_hyperthread_enabled\\\", \\\"false\\\", \\\"instance\\\", \\\"(.*)\\\"), 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000775525,\"lastEvaluation\":\"2024-04-12T22:36:47.965356087Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_node_labels\",\"query\":\"topk by (node) (1, cluster:nodes_roles) * on (node) group_right (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_node_openshift_io_os_id, label_kubernetes_io_arch, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) label_replace(cluster:cpu_core_hyperthreading, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000288696,\"lastEvaluation\":\"2024-04-12T22:36:47.966133642Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_usage_cores:sum\",\"query\":\"sum(1 - rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[2m]) * on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{pod=~\\\"node-exporter.+\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00054354,\"lastEvaluation\":\"2024-04-12T22:36:47.966921808Z\",\"type\":\"recording\"},{\"name\":\"cluster:hyperthread_enabled_nodes\",\"query\":\"kube_node_labels and on (node) (sum by (node, package, core) (label_replace(node_cpu_info, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")) == 2)\",\"labels\":{\"label_node_hyperthread_enabled\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000349047,\"lastEvaluation\":\"2024-04-12T22:36:47.964357608Z\",\"type\":\"recording\"},{\"name\":\"cluster:infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"infra\\\"})\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000170604,\"lastEvaluation\":\"2024-04-12T22:36:47.963852708Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolume_plugin_type_counts:sum\",\"query\":\"sum by (plugin_name, volume_mode) (pv_collector_total_pv_count{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000459179,\"lastEvaluation\":\"2024-04-12T22:36:47.995278127Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_resource_requests_storage_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000755684,\"lastEvaluation\":\"2024-04-12T22:36:47.985956067Z\",\"type\":\"recording\"},{\"name\":\"cluster:kubelet_volume_stats_used_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kubelet_volume_stats_used_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000401228,\"lastEvaluation\":\"2024-04-12T22:36:47.987359644Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (cluster:master_nodes and on (node) cluster:infra_nodes)\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000132182,\"lastEvaluation\":\"2024-04-12T22:36:47.964024441Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"label_node_role_kubernetes_io\":\"master\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000325696,\"lastEvaluation\":\"2024-04-12T22:36:47.963525342Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage:ratio\",\"query\":\"sum by (cluster) (container_memory_usage_bytes{container=\\\"\\\",pod!=\\\"\\\"}) / sum by (cluster) (machine_memory_bytes)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010134309,\"lastEvaluation\":\"2024-04-12T22:36:47.933709305Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage_bytes:sum\",\"query\":\"sum(node_memory_MemTotal_bytes{job=\\\"node-exporter\\\"} - node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000199454,\"lastEvaluation\":\"2024-04-12T22:36:47.967467268Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_instance_type_count:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) (cluster:master_nodes or on (node) kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000256065,\"lastEvaluation\":\"2024-04-12T22:36:47.985698522Z\",\"type\":\"recording\"},{\"name\":\"cluster:nodes_roles\",\"query\":\"cluster:master_infra_nodes or on (node) cluster:master_nodes or on (node) cluster:infra_nodes or on (node) max without (endpoint, instance, job, pod, service) (kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000198834,\"lastEvaluation\":\"2024-04-12T22:36:47.964157634Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:containers:sum\",\"query\":\"sum(max by (instance) (kubelet_containers_per_pod_count_sum))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000128573,\"lastEvaluation\":\"2024-04-12T22:36:47.994546103Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_node_ready:avg5m\",\"query\":\"avg_over_time((count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1)) / scalar(count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003232883,\"lastEvaluation\":\"2024-04-12T22:36:48.004967685Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_schedulable_node_ready_reachable:avg5m\",\"query\":\"avg_over_time((((count((max by (node) (up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1) and max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1) and min by (node) (kube_node_spec_unschedulable == 0))) / scalar(count(min by (node) (kube_node_spec_unschedulable == 0))))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006318392,\"lastEvaluation\":\"2024-04-12T22:36:47.998647403Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace=~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001252884,\"lastEvaluation\":\"2024-04-12T22:36:48.022030176Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:pods:terminal:workload:sum\",\"query\":\"count(count by (namespace, pod) (kube_pod_restart_policy{namespace!~\\\"openshift-.+\\\",type!=\\\"Always\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000127912,\"lastEvaluation\":\"2024-04-12T22:36:47.994417121Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:resources:sum\",\"query\":\"topk(500, max by (resource) (apiserver_storage_objects != -1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002758324,\"lastEvaluation\":\"2024-04-12T22:36:47.991656857Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:max:5m\",\"query\":\"max_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000062601,\"lastEvaluation\":\"2024-04-12T22:36:47.987296113Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:min:5m\",\"query\":\"min_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000095062,\"lastEvaluation\":\"2024-04-12T22:36:47.987200051Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace!~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001740804,\"lastEvaluation\":\"2024-04-12T22:36:48.02328452Z\",\"type\":\"recording\"},{\"name\":\"cluster:virt_platform_nodes:sum\",\"query\":\"count by (type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (sum by (instance, type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (virt_platform))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000180903,\"lastEvaluation\":\"2024-04-12T22:36:47.964707865Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_object_counts:sum\",\"query\":\"sum by (instance) (apiserver_storage_objects != -1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003892765,\"lastEvaluation\":\"2024-04-12T22:36:47.987761702Z\",\"type\":\"recording\"},{\"name\":\"kube_running_pod_ready\",\"query\":\"(max without (condition, container, endpoint, instance, job, service) (((kube_pod_status_ready{condition=\\\"false\\\"} == 1) * 0 or (kube_pod_status_ready{condition=\\\"true\\\"} == 1)) * on (pod, namespace) group_left () group by (pod, namespace) (kube_pod_status_phase{phase=~\\\"Running|Unknown|Pending\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013824628,\"lastEvaluation\":\"2024-04-12T22:36:48.008202298Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_cpu_usage:sum\",\"query\":\"sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=\\\"\\\",container!=\\\"POD\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007228534,\"lastEvaluation\":\"2024-04-12T22:36:47.92647816Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_memory_usage_bytes:sum\",\"query\":\"sum by (namespace) (container_memory_usage_bytes{container!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009425742,\"lastEvaluation\":\"2024-04-12T22:36:47.917049358Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_cores:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000163023,\"lastEvaluation\":\"2024-04-12T22:36:47.994675856Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_sockets:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (max by (node, package, label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000167663,\"lastEvaluation\":\"2024-04-12T22:36:47.994985912Z\",\"type\":\"recording\"},{\"name\":\"openshift:cpu_usage_cores:sum\",\"query\":\"cluster:cpu_usage_cores:sum - workload:cpu_usage_cores:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000148032,\"lastEvaluation\":\"2024-04-12T22:36:47.975989784Z\",\"type\":\"recording\"},{\"name\":\"openshift:memory_usage_bytes:sum\",\"query\":\"cluster:memory_usage_bytes:sum - workload:memory_usage_bytes:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000188714,\"lastEvaluation\":\"2024-04-12T22:36:47.985507768Z\",\"type\":\"recording\"},{\"name\":\"pod:container_cpu_usage:sum\",\"query\":\"sum by (pod, namespace) (rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010408544,\"lastEvaluation\":\"2024-04-12T22:36:47.895636169Z\",\"type\":\"recording\"},{\"name\":\"pod:container_fs_usage_bytes:sum\",\"query\":\"sum by (pod, namespace) (container_fs_usage_bytes{pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010997915,\"lastEvaluation\":\"2024-04-12T22:36:47.906047623Z\",\"type\":\"recording\"},{\"name\":\"profile:cluster_monitoring_operator_collection_profile:max\",\"query\":\"max by (profile) (cl{\"status\":\"success\",\"data\":{\"groups\":[{\"name\":\"CloudCredentialOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cloud-credential-operator-cloud-credential-operator-alerts-59f29d20-14e3-45ac-95b2-817cbe434ddc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorDeprovisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsDeprovisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest marked for deletion, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of CredentialsDeprovisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be cleaned up\",\"summary\":\"One or more CredentialsRequest CRs are unable to be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000117926,\"lastEvaluation\":\"2024-04-12T22:36:31.735304389Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorInsufficientCloudCreds\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"InsufficientCloudCreds\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator has determined that there are insufficient permissions to process one or more CredentialsRequest CRs. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of InsufficientCloudCreds set to True for more details.\",\"message\":\"Cluster's cloud credentials insufficient for minting or passthrough\",\"summary\":\"Problem with the available platform credentials.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000108965,\"lastEvaluation\":\"2024-04-12T22:36:31.735423516Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorProvisioningFailed\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"CredentialsProvisionFailure\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"While processing a CredentialsRequest, the Cloud Credential Operator encountered an issue. Check the conditions of all CredentialsRequets with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .stats.condition showing a condition type of CredentialsProvisionFailure set to True for more details on the issue.\",\"message\":\"CredentialsRequest(s) unable to be fulfilled\",\"summary\":\"One or more CredentialsRequest CRs are unable to be processed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000098825,\"lastEvaluation\":\"2024-04-12T22:36:31.735204354Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorStaleCredentials\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"StaleCredentials\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Cloud Credential Operator (CCO) has detected one or more stale CredentialsRequest CRs that need to be manually deleted. When the CCO is in Manual credentials mode, it will not automatially clean up stale CredentialsRequest CRs (that may no longer be necessary in the present version of OpenShift because it could involve needing to clean up manually created cloud resources. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of StaleCredentials set to True. Determine the appropriate steps to clean up/deprovision any previously provisioned cloud resources. Finally, delete the CredentialsRequest with an 'oc delete'.\",\"message\":\"1 or more credentials requests are stale and should be deleted. Check the status.conditions on CredentialsRequest CRs to identify the stale one(s).\",\"summary\":\"One or more CredentialsRequest CRs are stale and should be deleted.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000105386,\"lastEvaluation\":\"2024-04-12T22:36:31.735533862Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CloudCredentialOperatorTargetNamespaceMissing\",\"query\":\"cco_credentials_requests_conditions{condition=\\\"MissingTargetNamespace\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"At least one CredentialsRequest custom resource has specified in its .spec.secretRef.namespace field a namespace which does not presently exist. This means the Cloud Credential Operator in the openshift-cloud-credential-operator namespace cannot process the CredentialsRequest resource. Check the conditions of all CredentialsRequests with 'oc get credentialsrequest -A' to find any CredentialsRequest(s) with a .status.condition showing a condition type of MissingTargetNamespace set to True.\",\"message\":\"CredentialsRequest(s) pointing to non-existent namespace\",\"summary\":\"One ore more CredentialsRequest CRs are asking to save credentials to a non-existent namespace.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000150643,\"lastEvaluation\":\"2024-04-12T22:36:31.735039966Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00053507,\"lastEvaluation\":\"2024-04-12T22:36:31.735037066Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-machine-approver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-machine-approver-machineapprover-rules-02d4c4c8-9481-4482-a4eb-aa2196fb1abd.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineApproverMaxPendingCSRsReached\",\"query\":\"mapi_current_pending_csr \\u003e mapi_max_pending_csr\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of pending CertificateSigningRequests has exceeded the\\nmaximum threshold (current number of machine + 100). Check the\\npending CSRs to determine which machines need approval, also check\\nthat the nodelink controller is running in the openshift-machine-api\\nnamespace.\\n\",\"summary\":\"max pending CSRs threshold reached.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000264985,\"lastEvaluation\":\"2024-04-12T22:36:35.040054675Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000294385,\"lastEvaluation\":\"2024-04-12T22:36:35.039883304Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-tuning-operator.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-node-tuning-operator-node-tuning-operator-475016e5-fc0a-4af1-bd38-a45d26c2b85a.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NTODegraded\",\"query\":\"nto_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Node Tuning Operator is degraded. Review the \\\"node-tuning\\\" ClusterOperator object for further details.\",\"summary\":\"The Node Tuning Operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000087741,\"lastEvaluation\":\"2024-04-12T22:36:35.756300122Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NTOPodsNotReady\",\"query\":\"kube_pod_status_ready{condition=\\\"true\\\",namespace=\\\"openshift-cluster-node-tuning-operator\\\"} == 0\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.pod }} is not ready.\\nReview the \\\"Event\\\" objects in \\\"openshift-cluster-node-tuning-operator\\\" namespace for further details.\\n\",\"summary\":\"Pod {{ $labels.pod }} is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000291586,\"lastEvaluation\":\"2024-04-12T22:36:35.756006506Z\",\"type\":\"alerting\"},{\"name\":\"nto_custom_profiles:count\",\"query\":\"count by (_id) (nto_profile_calculated_total{profile!~\\\"openshift\\\",profile!~\\\"openshift-control-plane\\\",profile!~\\\"openshift-node\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000113402,\"lastEvaluation\":\"2024-04-12T22:36:35.756391654Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.00050883,\"lastEvaluation\":\"2024-04-12T22:36:35.755999396Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"SamplesOperator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-samples-operator-samples-operator-alerts-5c7cc52f-a9cb-47fe-a845-20e9f9a26649.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SamplesDegraded\",\"query\":\"openshift_samples_degraded_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples could not be deployed and the operator is degraded. Review the \\\"openshift-samples\\\" ClusterOperator object for further details.\\n\",\"summary\":\"Samples operator is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000093025,\"lastEvaluation\":\"2024-04-12T22:36:47.104513835Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesImagestreamImportFailing\",\"query\":\"sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m) \\u003e sum(openshift_samples_failed_imagestream_import_info)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports.  You can look at the \\\"openshift-samples\\\"\\nClusterOperator object for details. Most likely there are issues with the external image registry hosting\\nthe images that needs to be investigated.  Or you can consider marking samples operator Removed if you do not\\ncare about having sample imagestreams available.  The list of ImageStreams for which samples operator is\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is detecting problems with imagestream image imports\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000264614,\"lastEvaluation\":\"2024-04-12T22:36:47.10424756Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesInvalidConfig\",\"query\":\"openshift_samples_invalidconfig_info == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator has been given an invalid configuration.\\n\",\"summary\":\"Samples operator Invalid configuration\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077034,\"lastEvaluation\":\"2024-04-12T22:36:47.10460794Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingSecret\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_secret\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator cannot find the samples pull secret in the openshift namespace.\\n\",\"summary\":\"Samples operator is not able to find secret\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000078804,\"lastEvaluation\":\"2024-04-12T22:36:47.104685694Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesMissingTBRCredential\",\"query\":\"openshift_samples_invalidsecret_info{reason=\\\"missing_tbr_credential\\\"} == 1\",\"duration\":7200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The samples operator cannot find credentials for 'registry.redhat.io'. Many of the sample ImageStreams will fail to import unless the 'samplesRegistry' in the operator configuration is changed.\\n\",\"summary\":\"Samples operator is not able to find the credentials for registry\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000072294,\"lastEvaluation\":\"2024-04-12T22:36:47.104765228Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesRetriesMissingOnImagestreamImportFailing\",\"query\":\"sum(openshift_samples_failed_imagestream_import_info) \\u003e sum(openshift_samples_retry_imagestream_import_total) - sum(openshift_samples_retry_imagestream_import_total offset 30m)\",\"duration\":7200,\"labels\":{\"namespace\":\"openshift-cluster-samples-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Samples operator is detecting problems with imagestream image imports, and the periodic retries of those\\nimports are not occurring.  Contact support.  You can look at the \\\"openshift-samples\\\" ClusterOperator object\\nfor details. Most likely there are issues with the external image registry hosting the images that need to\\nbe investigated.  The list of ImageStreams that have failing imports are:\\n{{ range query \\\"openshift_samples_failed_imagestream_import_info \\u003e 0\\\" }}\\n  {{ .Labels.name }}\\n{{ end }}\\nHowever, the list of ImageStreams for which samples operator is retrying imports is:\\nretrying imports:\\n{{ range query \\\"openshift_samples_retry_imagestream_import_total \\u003e 0\\\" }}\\n   {{ .Labels.imagestreamname }}\\n{{ end }}\\n\",\"summary\":\"Samples operator is having problems with imagestream imports and its retries.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000430863,\"lastEvaluation\":\"2024-04-12T22:36:47.103814857Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SamplesTBRInaccessibleOnBoot\",\"query\":\"openshift_samples_tbr_inaccessible_info == 1\",\"duration\":172800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"One of two situations has occurred.  Either\\nsamples operator could not access 'registry.redhat.io' during its initial installation and it bootstrapped as removed.\\nIf this is expected, and stems from installing in a restricted network environment, please note that if you\\nplan on mirroring images associated with sample imagestreams into a registry available in your restricted\\nnetwork environment, and subsequently moving samples operator back to 'Managed' state, a list of the images\\nassociated with each image stream tag from the samples catalog is\\nprovided in the 'imagestreamtag-to-image' config map in the 'openshift-cluster-samples-operator' namespace to\\nassist the mirroring process.\\nOr, the use of allowed registries or blocked registries with global imagestream configuration will not allow\\nsamples operator to create imagestreams using the default image registry 'registry.redhat.io'.\\n\",\"summary\":\"Samples operator is not able to access the registry on boot\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000079234,\"lastEvaluation\":\"2024-04-12T22:36:47.104838392Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001115289,\"lastEvaluation\":\"2024-04-12T22:36:47.103804797Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"default-storage-classes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MultipleDefaultStorageClasses\",\"query\":\"max_over_time(default_storage_class_count[5m]) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster storage operator monitors all storage classes configured in the cluster\\nand checks there is not more than one default StorageClass configured.\\n\",\"message\":\"StorageClass count check is failing (there should not be more than one default StorageClass)\",\"summary\":\"More than one default StorageClass detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000251395,\"lastEvaluation\":\"2024-04-12T22:36:37.477974261Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000378831,\"lastEvaluation\":\"2024-04-12T22:36:37.477148942Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodStartupStorageOperationsFailing\",\"query\":\"increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status!=\\\"success\\\"}[5m]) \\u003e 0 and ignoring (status) increase(storage_operation_duration_seconds_count{operation_name=~\\\"volume_attach|volume_mount\\\",status=\\\"success\\\"}[5m]) == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Failing storage operation \\\"{{ $labels.operation_name }}\\\" of volume plugin {{ $labels.volume_plugin }} was preventing Pods on node {{ $labels.node }}\\nfrom starting for past 5 minutes.\\nPlease investigate Pods that are \\\"ContainerCreating\\\" on the node: \\\"oc get pod --field-selector=spec.nodeName={{ $labels.node }} --all-namespaces | grep ContainerCreating\\\".\\nEvents of the Pods should contain exact error message: \\\"oc describe pod -n \\u003cpod namespace\\u003e \\u003cpod name\\u003e\\\".\\n\",\"summary\":\"Pods can't start because {{ $labels.operation_name }} of volume plugin {{ $labels.volume_plugin }} is permanently failing on node {{ $labels.node }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.003662945,\"lastEvaluation\":\"2024-04-12T22:36:49.473150763Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003676175,\"lastEvaluation\":\"2024-04-12T22:36:49.473141023Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"storage-selinux.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-storage-operator-prometheus-541f39df-048b-4e6b-9fe9-2500a8922ecc.yaml\",\"rules\":[{\"name\":\"cluster:volume_manager_selinux_pod_context_mismatch_total\",\"query\":\"sum(volume_manager_selinux_pod_context_mismatch_warnings_total) + sum(volume_manager_selinux_pod_context_mismatch_errors_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000381297,\"lastEvaluation\":\"2024-04-12T22:36:28.076271377Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volume_context_mismatch_errors_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_errors_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000109212,\"lastEvaluation\":\"2024-04-12T22:36:28.076799817Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volume_context_mismatch_warnings_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volume_context_mismatch_warnings_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000142513,\"lastEvaluation\":\"2024-04-12T22:36:28.076655824Z\",\"type\":\"recording\"},{\"name\":\"cluster:volume_manager_selinux_volumes_admitted_total\",\"query\":\"sum by (volume_plugin) (volume_manager_selinux_volumes_admitted_total{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000286085,\"lastEvaluation\":\"2024-04-12T22:36:28.076910049Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001153521,\"lastEvaluation\":\"2024-04-12T22:36:28.075904055Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-operators\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-b7ec8a0e-ca37-48ff-91bd-48b44cd606d6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotEvaluateConditionalUpdates\",\"query\":\"max by (version, condition, status, reason) ((time() - cluster_version_conditional_update_condition_seconds{condition=\\\"Recommended\\\",status=\\\"Unknown\\\"}) \\u003e= 3600)\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to evaluate conditional update matches means that Cluster Version Operator cannot decide whether an update path is recommended or not.\",\"summary\":\"Cluster Version Operator cannot evaluate conditional update matches for {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000111616,\"lastEvaluation\":\"2024-04-12T22:36:55.503189163Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterNotUpgradeable\",\"query\":\"max by (namespace, name, condition, endpoint) (cluster_operator_conditions{condition=\\\"Upgradeable\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"In most cases, you will still be able to apply patch releases. Reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"Upgradeable\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0) (ne (len (label \\\"reason\\\" $value)) 0) }}{{label \\\"reason\\\" $value}}.{{end}}{{end}}{{end}} For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000250284,\"lastEvaluation\":\"2024-04-12T22:36:55.501908485Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDegraded\",\"query\":\"max by (namespace, name, reason) ((cluster_operator_conditions{condition=\\\"Degraded\\\",job=\\\"cluster-version-operator\\\"} or on (namespace, name) group by (namespace, name) (cluster_operator_up{job=\\\"cluster-version-operator\\\"})) == 1)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator is degraded because {{ $labels.reason }}, and the components it manages may have reduced quality of service.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has been degraded for 30 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000485326,\"lastEvaluation\":\"2024-04-12T22:36:55.502482036Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorDown\",\"query\":\"max by (namespace, name, reason) (cluster_operator_up{job=\\\"cluster-version-operator\\\"} == 0)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The {{ $labels.name }} operator may be down or disabled because {{ $labels.reason }}, and the components it manages may be unavailable or degraded.  Cluster upgrades may not complete. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator has not been available for 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000320467,\"lastEvaluation\":\"2024-04-12T22:36:55.502160489Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterOperatorFlapping\",\"query\":\"max by (namespace, name) (changes(cluster_operator_up{job=\\\"cluster-version-operator\\\"}[2m]) \\u003e 2)\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The  {{ $labels.name }} operator behavior might cause upgrades to be unstable. For more information refer to 'oc get -o yaml clusteroperator {{ $labels.name }}'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster operator up status is changing often.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000219901,\"lastEvaluation\":\"2024-04-12T22:36:55.502968272Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001326586,\"lastEvaluation\":\"2024-04-12T22:36:55.500807021Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-cluster-version-cluster-version-operator-b7ec8a0e-ca37-48ff-91bd-48b44cd606d6.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CannotRetrieveUpdates\",\"query\":\"max by (namespace) ((time() - cluster_version_operator_update_retrieval_timestamp_seconds) \\u003e= 3600 and ignoring (condition, name, reason) (cluster_operator_conditions{condition=\\\"RetrievedUpdates\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\",reason!=\\\"NoChannel\\\"}))\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason {{ with $cluster_operator_conditions := \\\"cluster_operator_conditions\\\" | query}}{{range $value := .}}{{if and (eq (label \\\"name\\\" $value) \\\"version\\\") (eq (label \\\"condition\\\" $value) \\\"RetrievedUpdates\\\") (eq (label \\\"endpoint\\\" $value) \\\"metrics\\\") (eq (value $value) 0.0)}}{{label \\\"reason\\\" $value}} {{end}}{{end}}{{end}}. For more information refer to `oc get clusterversion/version -o=jsonpath=\\\"{.status.conditions[?(.type=='RetrievedUpdates')]}{'\\\\n'}\\\"`{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Cluster version operator has not retrieved updates in {{ $value | humanizeDuration }}.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000241142,\"lastEvaluation\":\"2024-04-12T22:36:39.739623191Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterReleaseNotAccepted\",\"query\":\"max by (namespace, name, reason) (cluster_operator_conditions{condition=\\\"ReleaseAccepted\\\",endpoint=\\\"metrics\\\",name=\\\"version\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The desired cluster release has not been accepted because {{ $labels.reason }}, and the cluster will continue to reconcile an earlier release instead of moving towards that desired release.  For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"The desired cluster release has not been accepted for at least an hour.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000125806,\"lastEvaluation\":\"2024-04-12T22:36:39.73996714Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterVersionOperatorDown\",\"query\":\"absent(up{job=\\\"cluster-version-operator\\\"} == 1)\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-cluster-version\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair. {{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} For more information refer to {{ label \\\"url\\\" (first $console_url ) }}/k8s/cluster/projects/openshift-cluster-version.{{ end }}{{ end }}\",\"summary\":\"Cluster version operator has disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000237812,\"lastEvaluation\":\"2024-04-12T22:36:39.739383739Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UpdateAvailable\",\"query\":\"sum by (channel, namespace, upstream) (cluster_version_available_updates) \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"For more information refer to 'oc adm upgrade'{{ with $console_url := \\\"console_url\\\" | query }}{{ if ne (len (label \\\"url\\\" (first $console_url ) ) ) 0}} or {{ label \\\"url\\\" (first $console_url ) }}/settings/cluster/{{ end }}{{ end }}.\",\"summary\":\"Your upstream update recommendation service recommends you update your cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000098335,\"lastEvaluation\":\"2024-04-12T22:36:39.739867784Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000646182,\"lastEvaluation\":\"2024-04-12T22:36:39.738991737Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift/console-operator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-console-operator-cluster-monitoring-prometheus-rules-13a53b5a-afd9-4211-9680-fe10e32351a7.yaml\",\"rules\":[{\"name\":\"cluster:console_auth_login_failures_total:sum\",\"query\":\"sum by (reason) (console_auth_login_failures_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000116322,\"lastEvaluation\":\"2024-04-12T22:36:29.799875967Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_requests_total:sum\",\"query\":\"sum(console_auth_login_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000191814,\"lastEvaluation\":\"2024-04-12T22:36:29.799544241Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_login_successes_total:sum\",\"query\":\"sum by (role) (console_auth_login_successes_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000133083,\"lastEvaluation\":\"2024-04-12T22:36:29.799738254Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_auth_logout_requests_total:sum\",\"query\":\"sum by (reason) (console_auth_logout_requests_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000080572,\"lastEvaluation\":\"2024-04-12T22:36:29.799993909Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_customization_perspectives_info:max\",\"query\":\"max by (name, state) (console_customization_perspectives_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000078672,\"lastEvaluation\":\"2024-04-12T22:36:29.800314685Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_plugins_info:max\",\"query\":\"max by (name, state) (console_plugins_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000163453,\"lastEvaluation\":\"2024-04-12T22:36:29.800149772Z\",\"type\":\"recording\"},{\"name\":\"cluster:console_usage_users:max\",\"query\":\"max by (role) (console_usage_users)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000072861,\"lastEvaluation\":\"2024-04-12T22:36:29.800076061Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000905939,\"lastEvaluation\":\"2024-04-12T22:36:29.798943678Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-dns.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-dns-operator-dns-be119138-53ca-458e-a33e-816e88c6ca38.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CoreDNSErrorsHigh\",\"query\":\"(sum by (namespace) (rate(coredns_dns_responses_total{rcode=\\\"SERVFAIL\\\"}[5m])) / sum by (namespace) (rate(coredns_dns_responses_total[5m]))) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS is returning SERVFAIL for {{ $value | humanizePercentage }} of requests.\",\"summary\":\"CoreDNS serverfail\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000269855,\"lastEvaluation\":\"2024-04-12T22:36:51.98821851Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSHealthCheckSlow\",\"query\":\"histogram_quantile(0.95, sum by (instance, le) (rate(coredns_health_request_duration_seconds_bucket[5m]))) \\u003e 10\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"CoreDNS Health Checks are slowing down (instance {{ $labels.instance }})\",\"summary\":\"CoreDNS health checks\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000361857,\"lastEvaluation\":\"2024-04-12T22:36:51.987855743Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CoreDNSPanicking\",\"query\":\"increase(coredns_panics_total[10m]) \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} CoreDNS panics observed on {{ $labels.instance }}\",\"summary\":\"CoreDNS panic\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000244395,\"lastEvaluation\":\"2024-04-12T22:36:51.987609808Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001073007,\"lastEvaluation\":\"2024-04-12T22:36:51.983556814Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"etcd\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-bdafb11d-bf7f-4480-ae4e-7cbeddeea773.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdDatabaseHighFragmentationRatio\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) \\u003c 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes \\u003e 1.048576e+08\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size in use on instance {{ $labels.instance }} is {{ $value | humanizePercentage }} of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.\",\"runbook_url\":\"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation\",\"summary\":\"etcd database size in use is less than 50% of the actual allocated storage.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000273924,\"lastEvaluation\":\"2024-04-12T22:36:44.199758574Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdDatabaseQuotaLowSpace\",\"query\":\"(last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 \\u003e 95\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdDatabaseQuotaLowSpace.md\",\"summary\":\"etcd cluster database is running full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000176359,\"lastEvaluation\":\"2024-04-12T22:36:44.199396135Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdExcessiveDatabaseGrowth\",\"query\":\"predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4 * 60 * 60) \\u003e etcd_server_quota_backend_bytes\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": Predicting running out of disk space in the next four hours, based on write observations within the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.\",\"summary\":\"etcd cluster database growing very fast.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.0001838,\"lastEvaluation\":\"2024-04-12T22:36:44.199573524Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighCommitDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.25\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile commit durations {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile commit durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00036941,\"lastEvaluation\":\"2024-04-12T22:36:44.199025365Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000312917,\"lastEvaluation\":\"2024-04-12T22:36:44.198711528Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighFsyncDurations\",\"query\":\"histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.5\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile fsync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster 99th percentile fsync durations are too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00038715,\"lastEvaluation\":\"2024-04-12T22:36:44.198323418Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedProposals\",\"query\":\"rate(etcd_server_proposals_failed_total{job=~\\\".*etcd.*\\\"}[15m]) \\u003e 5\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} proposal failures within the last 30 minutes on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of proposal failures.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000140508,\"lastEvaluation\":\"2024-04-12T22:36:44.19818184Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMemberCommunicationSlow\",\"query\":\"histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~\\\".*etcd.*\\\"}[5m])) \\u003e 0.15\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member communication with {{ $labels.To }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster member communication is slow.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000718188,\"lastEvaluation\":\"2024-04-12T22:36:44.197462382Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdMembersDown\",\"query\":\"max without (endpoint) (sum without (instance) (up{job=~\\\".*etcd.*\\\"} == bool 0) or count without (To) (sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~\\\".*etcd.*\\\"}[2m])) \\u003e 0.01)) \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": members are down ({{ $value }}).\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdMembersDown.md\",\"summary\":\"etcd cluster members are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000461165,\"lastEvaluation\":\"2024-04-12T22:36:44.196853459Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdNoLeader\",\"query\":\"etcd_server_has_leader{job=~\\\".*etcd.*\\\"} == 0\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": member {{ $labels.instance }} has no leader.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdNoLeader.md\",\"summary\":\"etcd cluster has no leader.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000144798,\"lastEvaluation\":\"2024-04-12T22:36:44.197316204Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00318783,\"lastEvaluation\":\"2024-04-12T22:36:44.196846789Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-etcd-operator-etcd-prometheus-rules-bdafb11d-bf7f-4480-ae4e-7cbeddeea773.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"etcdGRPCRequestsSlow\",\"query\":\"histogram_quantile(0.99, sum without (grpc_type) (rate(grpc_server_handling_seconds_bucket{grpc_method!=\\\"Defragment\\\",grpc_type=\\\"unary\\\",job=\\\"etcd\\\"}[10m]))) \\u003e 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": 99th percentile of gRPC requests is {{ $value }}s on etcd instance {{ $labels.instance }} for {{ $labels.grpc_method }} method.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdGRPCRequestsSlow.md\",\"summary\":\"etcd grpc requests are slow\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.008333211,\"lastEvaluation\":\"2024-04-12T22:36:54.417336925Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 50\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighNumberOfFailedGRPCRequests.md\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.013154615,\"lastEvaluation\":\"2024-04-12T22:36:54.441269989Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfFailedGRPCRequests\",\"query\":\"(sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"etcd\\\"}[5m])) / (sum without (grpc_type, grpc_code) (rate(grpc_server_handled_total{job=\\\"etcd\\\"}[5m])) \\u003e 2 and on () (sum(cluster_infrastructure_provider{type!~\\\"ipi|BareMetal\\\"} == bool 1)))) * 100 \\u003e 10\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }}% of requests for {{ $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.\",\"summary\":\"etcd cluster has high number of failed grpc requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.014937774,\"lastEvaluation\":\"2024-04-12T22:36:54.425723935Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdHighNumberOfLeaderChanges\",\"query\":\"avg(changes(etcd_server_is_leader[10m])) \\u003e 5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"etcd cluster \\\"{{ $labels.job }}\\\": {{ $value }} average leader changes within the last 10 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.\",\"summary\":\"etcd cluster has high number of leader changes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000160483,\"lastEvaluation\":\"2024-04-12T22:36:54.454426404Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"etcdInsufficientMembers\",\"query\":\"sum without (instance, pod) (up{job=\\\"etcd\\\"} == bool 1 and etcd_server_has_leader{job=\\\"etcd\\\"} == bool 1) \\u003c ((count without (instance, pod) (up{job=\\\"etcd\\\"}) + 1) / 2)\",\"duration\":180,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"etcd is reporting fewer instances are available than are needed ({{ $value }}). When etcd does not have a majority of instances available the Kubernetes and OpenShift APIs will reject read and write requests and operations that preserve the health of workloads cannot be performed. This can occur when multiple control plane nodes are powered off or are unable to connect to each other via the network. Check that all control plane nodes are powered on and that network connections between each machine are functional.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdInsufficientMembers.md\",\"summary\":\"etcd is reporting that a majority of instances are unavailable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000322627,\"lastEvaluation\":\"2024-04-12T22:36:54.454587997Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.037339836,\"lastEvaluation\":\"2024-04-12T22:36:54.417245063Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imageregistry.operations.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-image-registry-rules-2e046cbc-5e3f-419b-9e59-085c6eb3d543.yaml\",\"rules\":[{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.Create\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000205973,\"lastEvaluation\":\"2024-04-12T22:36:27.142541754Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"BlobStore.ServeBlob\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"blob\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000348556,\"lastEvaluation\":\"2024-04-12T22:36:27.142191437Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Get\\\"}), \\\"operation\\\", \\\"get\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000202683,\"lastEvaluation\":\"2024-04-12T22:36:27.142749038Z\",\"type\":\"recording\"},{\"name\":\"imageregistry:operations_count:sum\",\"query\":\"label_replace(label_replace(sum by (operation) (imageregistry_request_duration_seconds_count{operation=\\\"ManifestService.Put\\\"}), \\\"operation\\\", \\\"create\\\", \\\"operation\\\", \\\"(.+)\\\"), \\\"resource_type\\\", \\\"manifest\\\", \\\"resource_type\\\", \\\"\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000142263,\"lastEvaluation\":\"2024-04-12T22:36:27.142952901Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001056526,\"lastEvaluation\":\"2024-04-12T22:36:27.141155686Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"imagestreams.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-image-registry-imagestreams-rules-62089bf6-dfc3-4319-af63-38d6191b31ea.yaml\",\"rules\":[{\"name\":\"imageregistry:imagestreamtags_count:sum\",\"query\":\"sum by (location, source) (image_registry_image_stream_tags_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000297936,\"lastEvaluation\":\"2024-04-12T22:36:29.517918454Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000318487,\"lastEvaluation\":\"2024-04-12T22:36:29.517397281Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress-to-route-controller.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-dfd4b459-b669-489c-9020-22defa539ee2.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"IngressWithoutClassName\",\"query\":\"openshift_ingress_to_route_controller_ingress_without_class_name == 1\",\"duration\":86400,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress {{ $labels.name }} is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"alerts\":[{\"labels\":{\"alertname\":\"IngressWithoutClassName\",\"container\":\"route-controller-manager\",\"endpoint\":\"https\",\"instance\":\"10.130.0.46:8443\",\"job\":\"route-controller-manager\",\"name\":\"test\",\"namespace\":\"e2e-test-router-stress-g64j9\",\"pod\":\"route-controller-manager-5bdb9f48f4-nbc4h\",\"service\":\"route-controller-manager\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is an Ingress with an unset IngressClassName for longer than one day.\",\"message\":\"Ingress test is missing the IngressClassName for 1 day.\",\"summary\":\"Ingress without IngressClassName for 1 day\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:21:56.656877779Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.00057098,\"lastEvaluation\":\"2024-04-12T22:36:26.658168642Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"UnmanagedRoutes\",\"query\":\"openshift_ingress_to_route_controller_route_with_unmanaged_owner == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when there is a Route owned by an unmanaged Ingress.\",\"message\":\"Route {{ $labels.name }} is owned by an unmanaged Ingress.\",\"summary\":\"Route owned by an Ingress no longer managed\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00055756,\"lastEvaluation\":\"2024-04-12T22:36:26.658742192Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00108014,\"lastEvaluation\":\"2024-04-12T22:36:26.657113115Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ingress-operator-ingress-operator-dfd4b459-b669-489c-9020-22defa539ee2.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HAProxyDown\",\"query\":\"haproxy_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"This alert fires when metrics report that HAProxy is down.\",\"message\":\"HAProxy metrics are reporting that HAProxy is down on pod {{ $labels.namespace }} / {{ $labels.pod }}\",\"summary\":\"HAProxy is down\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000117152,\"lastEvaluation\":\"2024-04-12T22:36:36.029997778Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HAProxyReloadFail\",\"query\":\"template_router_reload_failure == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when HAProxy fails to reload its configuration, which will result in the router not picking up recently created or modified routes.\",\"message\":\"HAProxy reloads are failing on {{ $labels.pod }}. Router is not respecting recently created or modified routes\",\"summary\":\"HAProxy reload failure\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000192924,\"lastEvaluation\":\"2024-04-12T22:36:36.029803404Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerDegraded\",\"query\":\"ingress_controller_conditions{condition=\\\"Degraded\\\"} == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController status is degraded.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\ndegraded: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is degraded\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000116962,\"lastEvaluation\":\"2024-04-12T22:36:36.0301163Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"IngressControllerUnavailable\",\"query\":\"ingress_controller_conditions{condition=\\\"Available\\\"} == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"This alert fires when the IngressController is not available.\",\"message\":\"The {{ $labels.namespace }}/{{ $labels.name }} ingresscontroller is\\nunavailable: {{ $labels.reason }}.\\n\",\"summary\":\"IngressController is unavailable\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000086392,\"lastEvaluation\":\"2024-04-12T22:36:36.030234183Z\",\"type\":\"alerting\"},{\"name\":\"cluster:openshift_route_info:tls_termination:sum\",\"query\":\"sum by (tls_termination) (openshift_route_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000177153,\"lastEvaluation\":\"2024-04-12T22:36:36.030713762Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:avg\",\"query\":\"avg(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000077671,\"lastEvaluation\":\"2024-04-12T22:36:36.030548629Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:max\",\"query\":\"max(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000088982,\"lastEvaluation\":\"2024-04-12T22:36:36.030458617Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:median\",\"query\":\"quantile(0.5, route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000085012,\"lastEvaluation\":\"2024-04-12T22:36:36.0306273Z\",\"type\":\"recording\"},{\"name\":\"cluster:route_metrics_controller_routes_per_shard:min\",\"query\":\"min(route_metrics_controller_routes_per_shard)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000134313,\"lastEvaluation\":\"2024-04-12T22:36:36.030321674Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001099001,\"lastEvaluation\":\"2024-04-12T22:36:36.029794314Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"insights\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-insights-insights-prometheus-rules-9ee17981-1562-4044-be51-c623d1ec6f10.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"InsightsDisabled\",\"query\":\"max without (job, pod, service, instance) (cluster_operator_conditions{condition=\\\"Disabled\\\",name=\\\"insights\\\"} == 1)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights operator is disabled. In order to enable Insights and benefit from recommendations specific to your cluster, please follow steps listed in the documentation: https://docs.openshift.com/container-platform/latest/support/remote_health_monitoring/enabling-remote-health-reporting.html\",\"summary\":\"Insights operator is disabled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000255843,\"lastEvaluation\":\"2024-04-12T22:36:26.980475055Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"InsightsRecommendationActive\",\"query\":\"insights_recommendation_active == 1\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Insights recommendation \\\"{{ $labels.description }}\\\" with total risk \\\"{{ $labels.total_risk }}\\\" was detected on the cluster. More information is available at {{ $labels.info_link }}.\",\"summary\":\"An Insights recommendation is active for this cluster.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077884,\"lastEvaluation\":\"2024-04-12T22:36:26.980878496Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"SimpleContentAccessNotAvailable\",\"query\":\"max without (job, pod, service, instance) (max_over_time(cluster_operator_conditions{condition=\\\"SCAAvailable\\\",name=\\\"insights\\\",reason=\\\"NotFound\\\"}[5m]) == 0)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-insights\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Simple content access (SCA) is not enabled. Once enabled, Insights Operator can automatically import the SCA certificates from Red Hat OpenShift Cluster Manager making it easier to use the content provided by your Red Hat subscriptions when creating container images. See https://docs.openshift.com/container-platform/latest/cicd/builds/running-entitled-builds.html for more information.\",\"summary\":\"Simple content access certificates are not available.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000144768,\"lastEvaluation\":\"2024-04-12T22:36:26.980732438Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000520169,\"lastEvaluation\":\"2024-04-12T22:36:26.980225537Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pre-release-lifecycle\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-api-usage-0d5178dd-4e23-49ca-884b-5d9c83f14fb4.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"APIRemovedInNextEUSReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.29\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next EUS version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next EUS version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.059891486,\"lastEvaluation\":\"2024-04-12T22:36:49.753094018Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"APIRemovedInNextReleaseInUse\",\"query\":\"group by (group, version, resource, removed_release) (apiserver_requested_deprecated_apis{removed_release=\\\"1.29\\\"}) * on (group, version, resource) group_left () sum by (group, version, resource) (rate(apiserver_request_total{system_client!=\\\"cluster-policy-controller\\\",system_client!=\\\"kube-controller-manager\\\"}[4h])) \\u003e 0\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version }}/{{ $labels.resource }} API might be necessary for a successful upgrade to the next cluster version with Kubernetes {{ $labels.removed_release }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml` to identify the workload.\",\"summary\":\"Deprecated API that will be removed in the next version is being used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.057387063,\"lastEvaluation\":\"2024-04-12T22:36:49.695703445Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.117296501,\"lastEvaluation\":\"2024-04-12T22:36:49.695692644Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-audit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-audit-errors-3528d707-66c1-425c-bbdb-459ae3d79e76.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AuditLogError\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_audit_error_total{apiserver=~\\\".+-apiserver\\\"}[5m])) / sum by (apiserver, instance) (rate(apiserver_audit_event_total{apiserver=~\\\".+-apiserver\\\"}[5m])) \\u003e 0\",\"duration\":60,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"An API Server had an error writing to an audit log.\",\"summary\":\"An API Server instance was unable to write audit logs. This could be\\ntriggered by the node running out of space, or a malicious actor\\ntampering with the audit logs.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000601002,\"lastEvaluation\":\"2024-04-12T22:36:35.38840949Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000720889,\"lastEvaluation\":\"2024-04-12T22:36:35.388292604Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"control-plane-cpu-utilization\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-cpu-utilization-7d5403fb-3bd5-4f24-ada6-427ad465fbde.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"Sustained high CPU utilization on a single control plane node, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000343136,\"lastEvaluation\":\"2024-04-12T22:36:36.635727642Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneCPU\",\"query\":\"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) \\u003e 90 and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Extreme CPU pressure can cause slow serialization and poor performance from the kube-apiserver and etcd. When this happens, there is a risk of clients seeing non-responsive API requests which are issued again causing even more CPU pressure. It can also cause failing liveness probes due to slow etcd responsiveness on the backend. If one kube-apiserver fails under this condition, chances are you will experience a cascade as the remaining kube-apiservers are also under-provisioned. To fix this, increase the CPU and memory on your control plane nodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization on a single control plane node is very high, more CPU pressure is likely to cause a failover; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000291696,\"lastEvaluation\":\"2024-04-12T22:36:36.635435026Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneCPU\",\"query\":\"sum(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[1m])) * 100) and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / count(kube_node_role{role=\\\"master\\\"}) \\u003e 60\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"On a multi-node cluster with three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes.\\nOn a single node OpenShift (SNO) cluster, this alert will also fire if the 2/3 of the CPU cores of the node are in use by any workload. This level of CPU utlization of an SNO cluster is probably not a problem under most circumstances, but high levels of utilization may result in degraded performance. To manage this alert or silence it in case of false positives see the following link:  https://docs.openshift.com/container-platform/latest/monitoring/managing-alerts.html\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md\",\"summary\":\"CPU utilization across all control plane nodes is more than 60% of the total available CPU. Control plane node outage may cause a cascading failure; increase available CPU.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000682083,\"lastEvaluation\":\"2024-04-12T22:36:36.634751593Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001333476,\"lastEvaluation\":\"2024-04-12T22:36:36.634740183Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"api-performance\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-performance-recording-rules-60c107f0-5c52-40c2-b772-4473cfac1e20.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000227105,\"lastEvaluation\":\"2024-04-12T22:36:54.152295946Z\",\"type\":\"recording\"},{\"name\":\"cluster:apiserver_tls_handshake_errors_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_tls_handshake_errors_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000176183,\"lastEvaluation\":\"2024-04-12T22:36:54.152524251Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.045000573,\"lastEvaluation\":\"2024-04-12T22:36:53.810314816Z\",\"type\":\"recording\"},{\"name\":\"code:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, code) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.04588809,\"lastEvaluation\":\"2024-04-12T22:36:53.855324249Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:1m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000426938,\"lastEvaluation\":\"2024-04-12T22:36:53.809526111Z\",\"type\":\"recording\"},{\"name\":\"component_resource:apiserver_request_terminations_total:rate:5m\",\"query\":\"sum by (apiserver, component, resource) (rate(apiserver_request_terminations_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000356677,\"lastEvaluation\":\"2024-04-12T22:36:53.809955289Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003394156,\"lastEvaluation\":\"2024-04-12T22:36:54.131202038Z\",\"type\":\"recording\"},{\"name\":\"filter:apiserver_request_filter_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, filter, le) (rate(apiserver_request_filter_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003666881,\"lastEvaluation\":\"2024-04-12T22:36:54.134598423Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_executing_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_executing_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000639063,\"lastEvaluation\":\"2024-04-12T22:36:54.124481797Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_current_inqueue_requests:sum\",\"query\":\"sum by (apiserver, flow_schema, priority_level) (apiserver_flowcontrol_current_inqueue_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000439359,\"lastEvaluation\":\"2024-04-12T22:36:54.125588749Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002956427,\"lastEvaluation\":\"2024-04-12T22:36:54.119568202Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum without (le) (flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001952648,\"lastEvaluation\":\"2024-04-12T22:36:54.122527549Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005841684,\"lastEvaluation\":\"2024-04-12T22:36:54.077450485Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level:apiserver_flowcontrol_request_wait_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_wait_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",execute=\\\"true\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005037797,\"lastEvaluation\":\"2024-04-12T22:36:54.083295749Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.015395089,\"lastEvaluation\":\"2024-04-12T22:36:54.088746644Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_le:apiserver_flowcontrol_request_execution_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, le) (rate(apiserver_flowcontrol_request_execution_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.015411169,\"lastEvaluation\":\"2024-04-12T22:36:54.104151323Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000234615,\"lastEvaluation\":\"2024-04-12T22:36:54.088337676Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_level_reason:apiserver_flowcontrol_rejected_requests_total:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason) (rate(apiserver_flowcontrol_rejected_requests_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000170883,\"lastEvaluation\":\"2024-04-12T22:36:54.088574531Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate1m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005023357,\"lastEvaluation\":\"2024-04-12T22:36:54.067724007Z\",\"type\":\"recording\"},{\"name\":\"flow_schema_priority_reason:apiserver_flowcontrol_request_queue_length_after_enqueue_bucket:rate5m\",\"query\":\"sum by (apiserver, flow_schema, priority_level, reason, le) (rate(apiserver_flowcontrol_request_queue_length_after_enqueue_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004696581,\"lastEvaluation\":\"2024-04-12T22:36:54.072750794Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003263723,\"lastEvaluation\":\"2024-04-12T22:36:54.143502616Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00255298,\"lastEvaluation\":\"2024-04-12T22:36:54.146768459Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate1m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002705812,\"lastEvaluation\":\"2024-04-12T22:36:54.138268044Z\",\"type\":\"recording\"},{\"name\":\"group_kind:apiserver_watch_events_total:rate5m\",\"query\":\"sum by (apiserver, group, kind) (rate(apiserver_watch_events_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002524289,\"lastEvaluation\":\"2024-04-12T22:36:54.140976037Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, group, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002969107,\"lastEvaluation\":\"2024-04-12T22:36:54.149323749Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000371917,\"lastEvaluation\":\"2024-04-12T22:36:53.808903419Z\",\"type\":\"recording\"},{\"name\":\"group_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, group, resource) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",code=\\\"429\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000246655,\"lastEvaluation\":\"2024-04-12T22:36:53.809277166Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, instance) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0046555,\"lastEvaluation\":\"2024-04-12T22:36:53.998565946Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.044223107,\"lastEvaluation\":\"2024-04-12T22:36:53.901220849Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.047218576,\"lastEvaluation\":\"2024-04-12T22:36:53.945452386Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.015738895,\"lastEvaluation\":\"2024-04-12T22:36:54.003766217Z\",\"type\":\"recording\"},{\"name\":\"instance:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, instance) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.016407038,\"lastEvaluation\":\"2024-04-12T22:36:54.019513082Z\",\"type\":\"recording\"},{\"name\":\"instance_request_kind:apiserver_current_inflight_requests:sum\",\"query\":\"sum by (apiserver, instance, request_kind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000538041,\"lastEvaluation\":\"2024-04-12T22:36:54.003225456Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.091631305,\"lastEvaluation\":\"2024-04-12T22:36:51.904027503Z\",\"type\":\"recording\"},{\"name\":\"list:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.102604299,\"lastEvaluation\":\"2024-04-12T22:36:51.996960149Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.32751045,\"lastEvaluation\":\"2024-04-12T22:36:52.725709768Z\",\"type\":\"recording\"},{\"name\":\"operation:etcd_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.56068537,\"lastEvaluation\":\"2024-04-12T22:36:53.053226938Z\",\"type\":\"recording\"},{\"name\":\"priority_level:apiserver_flowcontrol_request_concurrency_limit:sum\",\"query\":\"sum by (apiserver, priority_level) (apiserver_flowcontrol_request_concurrency_limit{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00046454,\"lastEvaluation\":\"2024-04-12T22:36:54.12512288Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.018830355,\"lastEvaluation\":\"2024-04-12T22:36:53.737931523Z\",\"type\":\"recording\"},{\"name\":\"read:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.019527649,\"lastEvaluation\":\"2024-04-12T22:36:53.756770698Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_longrunning_requests:sum\",\"query\":\"sum by (apiserver, resource) (apiserver_longrunning_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005881424,\"lastEvaluation\":\"2024-04-12T22:36:53.992679242Z\",\"type\":\"recording\"},{\"name\":\"resource:apiserver_storage_objects:max\",\"query\":\"max by (apiserver, resource) (apiserver_storage_objects{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002836445,\"lastEvaluation\":\"2024-04-12T22:36:54.152701434Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000346107,\"lastEvaluation\":\"2024-04-12T22:36:54.130565845Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_aborts_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_aborts_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000287345,\"lastEvaluation\":\"2024-04-12T22:36:54.130913382Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:1m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.283265351,\"lastEvaluation\":\"2024-04-12T22:36:51.386615443Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_duration_seconds_bucket:rate:5m\",\"query\":\"sum by (apiserver, resource, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.234128367,\"lastEvaluation\":\"2024-04-12T22:36:51.669889875Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.06807681,\"lastEvaluation\":\"2024-04-12T22:36:53.613921278Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.055914504,\"lastEvaluation\":\"2024-04-12T22:36:53.682007589Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.017559071,\"lastEvaluation\":\"2024-04-12T22:36:54.03592846Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_response_sizes_sum:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_response_sizes_sum{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.014222636,\"lastEvaluation\":\"2024-04-12T22:36:54.053495741Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate1m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002399647,\"lastEvaluation\":\"2024-04-12T22:36:54.126029407Z\",\"type\":\"recording\"},{\"name\":\"resource_verb:apiserver_selfrequest_total:rate5m\",\"query\":\"sum by (apiserver, resource, verb) (rate(apiserver_selfrequest_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002131721,\"lastEvaluation\":\"2024-04-12T22:36:54.128431954Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.202137845,\"lastEvaluation\":\"2024-04-12T22:36:52.287428993Z\",\"type\":\"recording\"},{\"name\":\"verb:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, verb, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb!~\\\"WATCH|WATCHLIST|PROXY\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.213206304,\"lastEvaluation\":\"2024-04-12T22:36:52.512496584Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate1m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.090115927,\"lastEvaluation\":\"2024-04-12T22:36:52.099571859Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_duration_seconds_bucket:rate5m\",\"query\":\"sum by (apiserver, le) (rate(apiserver_request_duration_seconds_bucket{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.093912566,\"lastEvaluation\":\"2024-04-12T22:36:52.193510476Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate1m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0160272,\"lastEvaluation\":\"2024-04-12T22:36:53.776307217Z\",\"type\":\"recording\"},{\"name\":\"write:apiserver_request_total:rate5m\",\"query\":\"sum by (apiserver) (rate(apiserver_request_total{apiserver=~\\\"openshift-apiserver|kube-apiserver|openshift-oauth-apiserver\\\",verb=~\\\"POST|PUT|PATCH|UPDATE|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.016552801,\"lastEvaluation\":\"2024-04-12T22:36:53.792341868Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":2.769699168,\"lastEvaluation\":\"2024-04-12T22:36:51.385841661Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-requests-in-flight\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-requests-6824c540-c854-48f2-9d56-c5eabf4a2b2f.yaml\",\"rules\":[{\"name\":\"cluster:apiserver_current_inflight_requests:sum:max_over_time:2m\",\"query\":\"max_over_time(sum by (apiserver, requestKind) (apiserver_current_inflight_requests{apiserver=~\\\"openshift-apiserver|kube-apiserver\\\"})[2m:])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000474705,\"lastEvaluation\":\"2024-04-12T22:36:27.870854345Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000492436,\"lastEvaluation\":\"2024-04-12T22:36:27.870839924Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-basic\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4f08eea7-5a77-4739-ba5f-4ebfa6a5c384.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1h) \\u003e (14.4 * 0.01) and sum(apiserver_request:burnrate5m) \\u003e (14.4 * 0.01)\",\"duration\":120,\"labels\":{\"long\":\"1h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"5m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000372347,\"lastEvaluation\":\"2024-04-12T22:36:25.801155645Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate6h) \\u003e (6 * 0.01) and sum(apiserver_request:burnrate30m) \\u003e (6 * 0.01)\",\"duration\":900,\"labels\":{\"long\":\"6h\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\",\"short\":\"30m\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000186373,\"lastEvaluation\":\"2024-04-12T22:36:25.801529542Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000451685,\"lastEvaluation\":\"2024-04-12T22:36:25.801132033Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4f08eea7-5a77-4739-ba5f-4ebfa6a5c384.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.015164194,\"lastEvaluation\":\"2024-04-12T22:36:38.558555321Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.010181167,\"lastEvaluation\":\"2024-04-12T22:36:38.588315118Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[30m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.009728989,\"lastEvaluation\":\"2024-04-12T22:36:38.548823922Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate30m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[30m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.007657447,\"lastEvaluation\":\"2024-04-12T22:36:38.598549263Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[5m])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.010208408,\"lastEvaluation\":\"2024-04-12T22:36:38.538611644Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate5m\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.005851462,\"lastEvaluation\":\"2024-04-12T22:36:38.606210031Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[6h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.014587073,\"lastEvaluation\":\"2024-04-12T22:36:38.573724735Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate6h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[6h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.009160747,\"lastEvaluation\":\"2024-04-12T22:36:38.612063442Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.121687879,\"lastEvaluation\":\"2024-04-12T22:36:39.048674253Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.130384218,\"lastEvaluation\":\"2024-04-12T22:36:38.918282635Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",subresource!=\\\"log\\\",verb!~\\\"LIST|WATCH|WATCHLIST|DELETECOLLECTION|PROXY|CONNECT\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.123482134,\"lastEvaluation\":\"2024-04-12T22:36:38.794796241Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.083649567,\"lastEvaluation\":\"2024-04-12T22:36:38.637102042Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (le, resource) (rate(apiserver_request_duration_seconds_bucket{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))) \\u003e 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.073791771,\"lastEvaluation\":\"2024-04-12T22:36:38.72098503Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.008557175,\"lastEvaluation\":\"2024-04-12T22:36:38.621226979Z\",\"type\":\"recording\"},{\"name\":\"code_resource:apiserver_request_total:rate5m\",\"query\":\"sum by (code, resource) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.007312438,\"lastEvaluation\":\"2024-04-12T22:36:38.629786694Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.631766988,\"lastEvaluation\":\"2024-04-12T22:36:38.538600094Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver-slos-extended\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-168fb4ca-48ed-4fa6-95a7-20c6c46562ef.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate1d) \\u003e (3 * 0.01) and sum(apiserver_request:burnrate2h) \\u003e (3 * 0.01)\",\"duration\":3600,\"labels\":{\"long\":\"1d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"2h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000321587,\"lastEvaluation\":\"2024-04-12T22:36:35.100389895Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPIErrorBudgetBurn\",\"query\":\"sum(apiserver_request:burnrate3d) \\u003e (1 * 0.01) and sum(apiserver_request:burnrate6h) \\u003e (1 * 0.01)\",\"duration\":10800,\"labels\":{\"long\":\"3d\",\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\",\"short\":\"6h\"},\"annotations\":{\"description\":\"The API server is burning too much error budget. This alert fires when too many requests are failing with high latency. Use the 'API Performance' monitoring dashboards to narrow down the request states and latency. The 'etcd' monitoring dashboards also provides metrics to help determine etcd stability and performance.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/KubeAPIErrorBudgetBurn.md\",\"summary\":\"The API server is burning too much error budget.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000424182,\"lastEvaluation\":\"2024-04-12T22:36:35.10077552Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001123449,\"lastEvaluation\":\"2024-04-12T22:36:35.100083524Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-apiserver.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-extended-168fb4ca-48ed-4fa6-95a7-20c6c46562ef.yaml\",\"rules\":[{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[1d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.019031292,\"lastEvaluation\":\"2024-04-12T22:36:31.518244181Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate1d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[1d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.010686038,\"lastEvaluation\":\"2024-04-12T22:36:31.553264243Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[2h])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.022732989,\"lastEvaluation\":\"2024-04-12T22:36:31.495505362Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate2h\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[2h]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.012240721,\"lastEvaluation\":\"2024-04-12T22:36:31.563953511Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) - ((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",scope=~\\\"resource|\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) or vector(0)) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"5\\\",scope=\\\"namespace\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])) + sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"30\\\",scope=\\\"cluster\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"LIST|GET\\\"}[3d])))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"LIST|GET\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"read\"},\"health\":\"ok\",\"evaluationTime\":0.015979289,\"lastEvaluation\":\"2024-04-12T22:36:31.537281113Z\",\"type\":\"recording\"},{\"name\":\"apiserver_request:burnrate3d\",\"query\":\"((sum by (cluster) (rate(apiserver_request_slo_duration_seconds_count{job=\\\"apiserver\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d])) - sum by (cluster) (rate(apiserver_request_slo_duration_seconds_bucket{job=\\\"apiserver\\\",le=\\\"1\\\",subresource!~\\\"proxy|attach|log|exec|portforward\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) + sum by (cluster) (rate(apiserver_request_total{code=~\\\"5..\\\",job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))) / sum by (cluster) (rate(apiserver_request_total{job=\\\"apiserver\\\",verb=~\\\"POST|PUT|PATCH|DELETE\\\"}[3d]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"verb\":\"write\"},\"health\":\"ok\",\"evaluationTime\":0.011007505,\"lastEvaluation\":\"2024-04-12T22:36:31.576198032Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.091709845,\"lastEvaluation\":\"2024-04-12T22:36:31.495498572Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-operator-kube-apiserver-operator-5005c833-ea1b-4666-ae08-3bea362dc2d3.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TechPreviewNoUpgrade\",\"query\":\"cluster_feature_set{name!=\\\"\\\",namespace=\\\"openshift-kube-apiserver-operator\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster has enabled Technology Preview features that cannot be undone and will prevent upgrades. The TechPreviewNoUpgrade feature set is not recommended on production clusters.\",\"summary\":\"Cluster has enabled tech preview features that will prevent upgrades.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002395806,\"lastEvaluation\":\"2024-04-12T22:36:45.48300113Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002119112,\"lastEvaluation\":\"2024-04-12T22:36:45.482834302Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"pod-security-violation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-podsecurity-be42b924-fa88-4cdc-b097-1e5e3015f2ef.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace!=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created in namespace \\\"{{ $labels.ocp_namespace }}\\\" but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads in platform namespaces of the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000136227,\"lastEvaluation\":\"2024-04-12T22:36:27.163648332Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodSecurityViolation\",\"query\":\"sum by (policy_level, ocp_namespace) (increase(pod_security_evaluations_total{decision=\\\"deny\\\",mode=\\\"audit\\\",ocp_namespace=\\\"\\\",resource=\\\"pod\\\"}[1d])) \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-kube-apiserver\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity \\\"{{ $labels.policy_level }}\\\" profile defined by its namespace either via the cluster-wide configuration (which triggers on a \\\"restricted\\\" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.\",\"summary\":\"One or more workloads users created in the cluster don't match their Pod Security profile\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000253205,\"lastEvaluation\":\"2024-04-12T22:36:27.163347548Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000380587,\"lastEvaluation\":\"2024-04-12T22:36:27.163343618Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-controller-manager-operator-kube-controller-manager-operator-59a05266-d32f-408a-b2c9-134a0aa47a1f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"GarbageCollectorSyncFailed\",\"query\":\"rate(garbagecollector_controller_resources_sync_error_total[5m]) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Garbage Collector had a problem with syncing and monitoring the available resources. Please see KubeControllerManager logs for more details.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/GarbageCollectorSyncFailed.md\",\"summary\":\"There was a problem with syncing the resources for garbage collection.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000191034,\"lastEvaluation\":\"2024-04-12T22:36:26.687631783Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeControllerManagerDown\",\"query\":\"absent(up{job=\\\"kube-controller-manager\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-controller-manager\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeControllerManager has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/KubeControllerManagerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000311836,\"lastEvaluation\":\"2024-04-12T22:36:26.686338099Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetAtLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy and on (namespace, poddisruptionbudget) kube_poddisruptionbudget_status_expected_pods \\u003e 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The pod disruption budget is at the minimum disruptions allowed level. The number of current healthy pods is equal to the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetAtLimit.md\",\"summary\":\"The pod disruption budget is preventing further disruption to pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000466764,\"lastEvaluation\":\"2024-04-12T22:36:26.686652636Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PodDisruptionBudgetLimit\",\"query\":\"max by (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_current_healthy \\u003c kube_poddisruptionbudget_status_desired_healthy)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The pod disruption budget is below the minimum disruptions allowed level and is not satisfied. The number of current healthy pods is less than the desired healthy pods.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-controller-manager-operator/PodDisruptionBudgetLimit.md\",\"summary\":\"The pod disruption budget registers insufficient amount of pods.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000365197,\"lastEvaluation\":\"2024-04-12T22:36:26.687264646Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00158003,\"lastEvaluation\":\"2024-04-12T22:36:26.686246467Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-version\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-14fb49f3-f1fb-4874-93f5-bc12c6437bcf.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeSchedulerDown\",\"query\":\"absent(up{job=\\\"scheduler\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-kube-scheduler\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeScheduler has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-scheduler-operator/KubeSchedulerDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000321926,\"lastEvaluation\":\"2024-04-12T22:36:31.339316817Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000336017,\"lastEvaluation\":\"2024-04-12T22:36:31.339306417Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"scheduler-legacy-policy-deprecated\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-scheduler-operator-kube-scheduler-operator-14fb49f3-f1fb-4874-93f5-bc12c6437bcf.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SchedulerLegacyPolicySet\",\"query\":\"cluster_legacy_scheduler_policy \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The scheduler is currently configured to use a legacy scheduler policy API. Use of the policy API is deprecated and removed in 4.10.\",\"summary\":\"Legacy scheduler policy API in use by the scheduler.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000216224,\"lastEvaluation\":\"2024-04-12T22:36:43.443324055Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000229705,\"lastEvaluation\":\"2024-04-12T22:36:43.443314645Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-api-operator-metrics-collector-up\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineAPIOperatorMetricsCollectionFailing\",\"query\":\"mapi_mao_collector_up == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"For more details:  oc logs \\u003cmachine-api-operator-pod-name\\u003e -n openshift-machine-api\",\"summary\":\"machine api operator metrics collection is failing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000266805,\"lastEvaluation\":\"2024-04-12T22:36:53.975148282Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000279985,\"lastEvaluation\":\"2024-04-12T22:36:53.975138542Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-health-check-unterminated-short-circuit\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineHealthCheckUnterminatedShortCircuit\",\"query\":\"mapi_machinehealthcheck_short_circuit == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\\nthe status of machines in the cluster.\\n\",\"summary\":\"machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000221112,\"lastEvaluation\":\"2024-04-12T22:36:51.928398519Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000233643,\"lastEvaluation\":\"2024-04-12T22:36:51.928389159Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-not-yet-deleted\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineNotYetDeleted\",\"query\":\"sum by (name, namespace) (avg_over_time(mapi_machine_created_timestamp_seconds{phase=\\\"Deleting\\\"}[15m])) \\u003e 0\",\"duration\":21600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine is not properly deleting, this may be due to a configuration issue with the\\ninfrastructure provider, or because workloads on the node have PodDisruptionBudgets or\\nlong termination periods which are preventing deletion.\\n\",\"summary\":\"machine {{ $labels.name }} has been in Deleting phase for more than 6 hours\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000272524,\"lastEvaluation\":\"2024-04-12T22:36:44.941617306Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000285856,\"lastEvaluation\":\"2024-04-12T22:36:44.941608115Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-with-no-running-phase\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithNoRunningPhase\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds{phase!~\\\"Running|Deleting\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The machine has been without a Running or Deleting phase for more than 60 minutes.\\nThe machine may not have been provisioned properly from the infrastructure provider, or\\nit might have issues with CertificateSigningRequests being approved.\\n\",\"summary\":\"machine {{ $labels.name }} is in phase: {{ $labels.phase }}\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000326337,\"lastEvaluation\":\"2024-04-12T22:36:31.397113273Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000264484,\"lastEvaluation\":\"2024-04-12T22:36:31.396641077Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"machine-without-valid-node-ref\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-bc318f18-4ff5-4101-a665-0b2bf3b324cb.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MachineWithoutValidNode\",\"query\":\"sum by (name, namespace) (mapi_machine_created_timestamp_seconds unless on (node) kube_node_info) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"If the machine never became a node, you should diagnose the machine related failures.\\nIf the node was deleted from the API, you may delete the machine if appropriate.\\n\",\"summary\":\"machine {{ $labels.name }} does not have valid node reference\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000335148,\"lastEvaluation\":\"2024-04-12T22:36:45.843108201Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000363587,\"lastEvaluation\":\"2024-04-12T22:36:45.836234369Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-drain-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCDrainError\",\"query\":\"mcc_drain_err \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Drain failed on {{ $labels.exported_node }} , updates may be blocked. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Alerts the user to a failed node drain. Always triggers when the failure happens one or more times.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000240684,\"lastEvaluation\":\"2024-04-12T22:36:26.126555741Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000254695,\"lastEvaluation\":\"2024-04-12T22:36:26.126545581Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcc-pool-alert\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCCPoolAlert\",\"query\":\"mcc_pool_alert \\u003e 0\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Node {{ $labels.exported_node }} has triggered a pool alert due to a label change. For more details check MachineConfigController pod logs: oc logs -f -n {{ $labels.namespace }} machine-config-controller-xxxxx -c machine-config-controller\",\"summary\":\"Triggers when nodes in a pool have overlapping labels such as master, worker, and a custom label therefore a choice must be made as to which is honored.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000245005,\"lastEvaluation\":\"2024-04-12T22:36:45.350111834Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000257436,\"lastEvaluation\":\"2024-04-12T22:36:45.350103014Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"os-image-override.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-controller-0a9fe235-e2f8-4723-9e47-c15a4e5c9f55.yaml\",\"rules\":[{\"name\":\"os_image_url_override:sum\",\"query\":\"sum(os_image_url_override)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000268834,\"lastEvaluation\":\"2024-04-12T22:36:55.674983934Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000281355,\"lastEvaluation\":\"2024-04-12T22:36:55.674974503Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"extremely-high-individual-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ExtremelyHighIndividualControlPlaneMemory\",\"query\":\"(1 - sum by (instance) (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum by (instance) (node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 90\",\"duration\":2700,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The memory utilization per instance within control plane nodes influence the stability, and responsiveness of the cluster. This can lead to cluster instability and slow responses from kube-apiserver or failing requests specially on etcd. Moreover, OOM kill is expected which negatively influences the pod scheduling. If this happens on container level, the descheduler will not be able to detect it, as it works on the pod level. To fix this, increase memory of the affected node of control plane nodes.\",\"summary\":\"Extreme memory utilization per node within control plane nodes is extremely high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000650663,\"lastEvaluation\":\"2024-04-12T22:36:39.133743062Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000675846,\"lastEvaluation\":\"2024-04-12T22:36:39.132725259Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"high-overall-control-plane-memory\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"HighOverallControlPlaneMemory\",\"query\":\"(1 - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\")) / sum(node_memory_MemTotal_bytes and on (instance) label_replace(kube_node_role{role=\\\"master\\\"}, \\\"instance\\\", \\\"$1\\\", \\\"node\\\", \\\"(.+)\\\"))) * 100 \\u003e 60\",\"duration\":3600,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Given three control plane nodes, the overall memory utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the kube-apiserver and etcd may be slow to respond. To fix this, increase memory of the control plane nodes.\",\"summary\":\"Memory utilization across all control plane nodes is high, and could impact responsiveness and stability.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000830794,\"lastEvaluation\":\"2024-04-12T22:36:37.080318668Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000622492,\"lastEvaluation\":\"2024-04-12T22:36:37.079547936Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-kubelet-health-state-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeletHealthState\",\"query\":\"mcd_kubelet_state \\u003e 2\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet health failure threshold reached\",\"summary\":\"This keeps track of Kubelet health failures, and tallys them. The warning is triggered if 2 or more failures occur.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000346499,\"lastEvaluation\":\"2024-04-12T22:36:36.713249675Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000241255,\"lastEvaluation\":\"2024-04-12T22:36:36.712959868Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-pivot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDPivotError\",\"query\":\"mcd_pivot_errors_total \\u003e 0\",\"duration\":120,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user when an error is detected upon pivot. This triggers if the pivot errors are above zero for 2 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000243424,\"lastEvaluation\":\"2024-04-12T22:36:26.054822042Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000256836,\"lastEvaluation\":\"2024-04-12T22:36:26.054811371Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"mcd-reboot-error\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"MCDRebootError\",\"query\":\"mcd_reboots_failed_total \\u003e 0\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Reboot failed on {{ $labels.node }} , update may be blocked. For more details:  oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon \",\"summary\":\"Alerts the user that a node failed to reboot one or more times over a span of 5 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000254155,\"lastEvaluation\":\"2024-04-12T22:36:54.372174709Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000267525,\"lastEvaluation\":\"2024-04-12T22:36:54.372164859Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"system-memory-exceeds-reservation\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-config-operator-machine-config-daemon-d66c097a-4989-4d58-a5b4-7e752e09c082.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"SystemMemoryExceedsReservation\",\"query\":\"sum by (node) (container_memory_rss{id=\\\"/system.slice\\\"}) \\u003e ((sum by (node) (kube_node_status_capacity{resource=\\\"memory\\\"} - kube_node_status_allocatable{resource=\\\"memory\\\"})) * 0.95)\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-machine-config-operator\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).\",\"summary\":\"Alerts the user when, for 15 miutes, a specific node is using more memory than is reserved\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000574741,\"lastEvaluation\":\"2024-04-12T22:36:44.345946605Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000548709,\"lastEvaluation\":\"2024-04-12T22:36:44.345632125Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"operator.marketplace.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-marketplace-marketplace-alert-rules-714df196-fc7d-484a-8308-076cc71239c1.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"OperatorHubSourceError\",\"query\":\"catalogsource_ready{exported_namespace=\\\"openshift-marketplace\\\"} == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Operators shipped via the {{ $labels.name }} source are not available for installation until the issue is fixed. Operators already installed from this source will not receive updates until issue is fixed. Inspect the status of the pod owned by {{ $labels.name }} source in the openshift-marketplace namespace (oc -n openshift-marketplace get pods -l olm.catalogSource={{ $labels.name }}) to diagnose and repair.\",\"summary\":\"The {{ $labels.name }} source is in non-ready state for more than 10 minutes.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000205474,\"lastEvaluation\":\"2024-04-12T22:36:26.137690192Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000213214,\"lastEvaluation\":\"2024-04-12T22:36:26.137684882Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"alertmanager.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-alertmanager-main-rules-b89ef7fc-8e25-4a61-90d5-f3ea1f8d7983.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterDown\",\"query\":\"(count by (namespace, service) (avg_over_time(up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c 0.5) / count by (namespace, service) (up{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) \\u003e= 0.5\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of Alertmanager instances within the {{$labels.job}} cluster have been up for less than half of the last 5m.\",\"summary\":\"Half or more of the Alertmanager instances within the same cluster are down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000275936,\"lastEvaluation\":\"2024-04-12T22:36:40.32919809Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerClusterFailedToSendAlerts\",\"query\":\"min by (namespace, service, integration) (rate(alertmanager_notifications_failed_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{integration=~\\\".*\\\",job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The minimum notification failure rate to {{ $labels.integration }} sent from any instance in the {{$labels.job}} cluster is {{ $value | humanizePercentage }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerClusterFailedToSendAlerts.md\",\"summary\":\"All Alertmanager instances in a cluster failed to send notifications to a critical integration.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000824656,\"lastEvaluation\":\"2024-04-12T22:36:40.328170879Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerConfigInconsistent\",\"query\":\"count by (namespace, service) (count_values by (namespace, service) (\\\"config_hash\\\", alertmanager_config_hash{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"})) != 1\",\"duration\":1200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager instances within the {{$labels.job}} cluster have different configurations.\",\"summary\":\"Alertmanager instances within the same cluster have different configurations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000199453,\"lastEvaluation\":\"2024-04-12T22:36:40.328996846Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedReload\",\"query\":\"max_over_time(alertmanager_config_last_reload_successful{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Configuration has failed to load for {{ $labels.namespace }}/{{ $labels.pod}}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedReload.md\",\"summary\":\"Reloading an Alertmanager configuration has failed.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000355179,\"lastEvaluation\":\"2024-04-12T22:36:40.326475027Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerFailedToSendAlerts\",\"query\":\"(rate(alertmanager_notifications_failed_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) / ignoring (reason) group_left () rate(alertmanager_notifications_total{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m])) \\u003e 0.01\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} failed to send {{ $value | humanizePercentage }} of notifications to {{ $labels.integration }}.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedToSendAlerts.md\",\"summary\":\"An Alertmanager instance failed to send notifications.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000700037,\"lastEvaluation\":\"2024-04-12T22:36:40.327184894Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"AlertmanagerMembersInconsistent\",\"query\":\"max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]) \\u003c on (namespace, service) group_left () count by (namespace, service) (max_over_time(alertmanager_cluster_members{job=~\\\"alertmanager-main|alertmanager-user-workload\\\"}[5m]))\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} has only found {{ $value }} members of the {{$labels.job}} cluster.\",\"summary\":\"A member of an Alertmanager cluster has not found all other cluster members.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000351658,\"lastEvaluation\":\"2024-04-12T22:36:40.326831955Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003114101,\"lastEvaluation\":\"2024-04-12T22:36:40.326362195Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"apiserver-list-watch.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"apiserver_list_watch_request_success_total:rate:sum\",\"query\":\"sum by (verb) (rate(apiserver_request_total{code=~\\\"2..\\\",verb=~\\\"LIST|WATCH\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011091991,\"lastEvaluation\":\"2024-04-12T22:36:49.03612487Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.01343978,\"lastEvaluation\":\"2024-04-12T22:36:49.035981344Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"Watchdog\",\"query\":\"vector(1)\",\"duration\":0,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"alerts\":[{\"labels\":{\"alertname\":\"Watchdog\",\"namespace\":\"openshift-monitoring\",\"severity\":\"none\"},\"annotations\":{\"description\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\\n\",\"summary\":\"An alert that should always be firing to certify that Alertmanager is working properly.\"},\"state\":\"firing\",\"activeAt\":\"2024-04-12T21:38:00.04805628Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.00036572,\"lastEvaluation\":\"2024-04-12T22:36:30.049373349Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000325186,\"lastEvaluation\":\"2024-04-12T22:36:30.049005388Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"count:up0\",\"query\":\"count without (instance, pod, node) (up == 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000584401,\"lastEvaluation\":\"2024-04-12T22:36:30.562035441Z\",\"type\":\"recording\"},{\"name\":\"count:up1\",\"query\":\"count without (instance, pod, node) (up == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001188013,\"lastEvaluation\":\"2024-04-12T22:36:30.560845548Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.001786285,\"lastEvaluation\":\"2024-04-12T22:36:30.560835887Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-prometheus-node-recording.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:node_cpu:ratio\",\"query\":\"cluster:node_cpu:sum_rate5m / count(sum by (instance, cpu) (node_cpu_seconds_total))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001001909,\"lastEvaluation\":\"2024-04-12T22:36:31.717688757Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:sum_rate5m\",\"query\":\"sum(rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000501319,\"lastEvaluation\":\"2024-04-12T22:36:31.717186508Z\",\"type\":\"recording\"},{\"name\":\"instance:node_cpu:rate:sum\",\"query\":\"sum by (instance) (rate(node_cpu_seconds_total{mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000744524,\"lastEvaluation\":\"2024-04-12T22:36:31.715888793Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_receive_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000298946,\"lastEvaluation\":\"2024-04-12T22:36:31.716635227Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_transmit_bytes:rate:sum\",\"query\":\"sum by (instance) (rate(node_network_transmit_bytes_total[3m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000250275,\"lastEvaluation\":\"2024-04-12T22:36:31.716935283Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002811523,\"lastEvaluation\":\"2024-04-12T22:36:31.715881853Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-recurring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:usage:workload:capacity_physical_cpu_core_seconds\",\"query\":\"sum_over_time(workload:capacity_physical_cpu_cores:sum[30s:1s]) + ((cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) or (absent(cluster:usage:workload:capacity_physical_cpu_core_seconds offset 25s) * 0))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000427389,\"lastEvaluation\":\"2024-04-12T22:36:34.377895604Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000439449,\"lastEvaluation\":\"2024-04-12T22:36:34.377886824Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-network\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeNetworkInterfaceFlapping\",\"query\":\"changes(node_network_up{device!~\\\"veth.+|tunbr\\\",job=\\\"node-exporter\\\"}[2m]) \\u003e 2\",\"duration\":120,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Network interface \\\"{{ $labels.device }}\\\" changing its up status often on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}\",\"summary\":\"Network interface is often changing its status\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000479155,\"lastEvaluation\":\"2024-04-12T22:36:25.834301237Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000651173,\"lastEvaluation\":\"2024-04-12T22:36:25.833252363Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-build.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"openshift:build_by_strategy:sum\",\"query\":\"sum by (strategy) (openshift_build_status_phase_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000352997,\"lastEvaluation\":\"2024-04-12T22:36:39.546313771Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000366027,\"lastEvaluation\":\"2024-04-12T22:36:39.546304101Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-etcd-telemetry.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_backend_commit_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000471499,\"lastEvaluation\":\"2024-04-12T22:36:46.87614077Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000474385,\"lastEvaluation\":\"2024-04-12T22:36:46.874714269Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000240762,\"lastEvaluation\":\"2024-04-12T22:36:46.874471787Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum\",\"query\":\"sum by (instance) (etcd_mvcc_db_total_size_in_use_in_bytes{job=\\\"etcd\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000215515,\"lastEvaluation\":\"2024-04-12T22:36:46.875923125Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{job=\\\"etcd\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000687606,\"lastEvaluation\":\"2024-04-12T22:36:46.875190305Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002345756,\"lastEvaluation\":\"2024-04-12T22:36:46.874268973Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-general.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"TargetDown\",\"query\":\"100 * ((1 - sum by (job, namespace, service) (up and on (namespace, pod) kube_pod_info) / count by (job, namespace, service) (up and on (namespace, pod) kube_pod_info)) or (count by (job, namespace, service) (up == 0) / count by (job, namespace, service) (up))) \\u003e 10\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ printf \\\"%.4g\\\" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.\",\"summary\":\"Some targets were not reachable from the monitoring server for an extended period of time.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00725452,\"lastEvaluation\":\"2024-04-12T22:36:49.131392564Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00726747,\"lastEvaluation\":\"2024-04-12T22:36:49.131383473Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-ingress.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:ingress_controller_aws_nlb_active:sum\",\"query\":\"sum(ingress_controller_aws_nlb_active) or vector(0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000117253,\"lastEvaluation\":\"2024-04-12T22:36:47.632569255Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_in:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_in_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000150232,\"lastEvaluation\":\"2024-04-12T22:36:47.628183781Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_bytes_out:rate5m:sum\",\"query\":\"sum(rate(haproxy_frontend_bytes_out_total[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000159203,\"lastEvaluation\":\"2024-04-12T22:36:47.628335493Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:ingress_frontend_connections:sum\",\"query\":\"sum(haproxy_frontend_current_sessions)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000108811,\"lastEvaluation\":\"2024-04-12T22:36:47.628495827Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace=~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00157298,\"lastEvaluation\":\"2024-04-12T22:36:47.629900964Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace=~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001092682,\"lastEvaluation\":\"2024-04-12T22:36:47.631475284Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_error:fraction5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{code!~\\\"2xx|1xx|4xx|3xx\\\",exported_namespace!~\\\"openshift-.*\\\"}[5m]) \\u003e 0)) / sum(max without (service, endpoint, container, pod, job, namespace) (increase(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000785795,\"lastEvaluation\":\"2024-04-12T22:36:47.628605789Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:ingress_request_total:irate5m\",\"query\":\"sum(max without (service, endpoint, container, pod, job, namespace) (irate(haproxy_server_http_responses_total{exported_namespace!~\\\"openshift-.*\\\"}[5m]))) or absent(__does_not_exist__) * 0\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00050599,\"lastEvaluation\":\"2024-04-12T22:36:47.629393424Z\",\"type\":\"recording\"},{\"name\":\"code:cluster:ingress_http_request_count:rate5m:sum\",\"query\":\"sum by (code) (rate(haproxy_server_http_responses_total[5m]) \\u003e 0)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001585241,\"lastEvaluation\":\"2024-04-12T22:36:47.62659673Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006102148,\"lastEvaluation\":\"2024-04-12T22:36:47.62658617Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-kubernetes.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"state\":\"firing\",\"name\":\"AlertmanagerReceiversNotConfigured\",\"query\":\"cluster:alertmanager_integrations:max == 0\",\"duration\":600,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"alerts\":[{\"labels\":{\"alertname\":\"AlertmanagerReceiversNotConfigured\",\"namespace\":\"openshift-monitoring\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.\",\"summary\":\"Receivers (notification integrations) are not configured on Alertmanager\"},\"state\":\"firing\",\"activeAt\":\"2024-04-12T21:39:17.894182481Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000165894,\"lastEvaluation\":\"2024-04-12T22:36:47.996157584Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ClusterMonitoringOperatorReconciliationErrors\",\"query\":\"max_over_time(cluster_monitoring_operator_last_reconciliation_successful[5m]) == 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors are occurring during reconciliation cycles. Inspect the cluster-monitoring-operator log for potential root causes.\",\"summary\":\"Cluster Monitoring Operator is experiencing unexpected reconciliation errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000077241,\"lastEvaluation\":\"2024-04-12T22:36:47.996079603Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentReplicasMismatch\",\"query\":\"(((kube_deployment_spec_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_deployment_status_replicas_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_deployment_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)) * on () group_left () cluster:control_plane:all_nodes_ready) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes. This indicates that cluster infrastructure is unable to start or restart the necessary components. This most often occurs when one or more nodes are down or partioned from the cluster, or a fault occurs on the node that prevents the workload from starting. In rare cases this may indicate a new version of a cluster component cannot start due to a bug or configuration error. Assess the pods for this deployment to verify they are running on healthy nodes and then contact support.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeDeploymentReplicasMismatch.md\",\"summary\":\"Deployment has not matched the expected number of replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002321535,\"lastEvaluation\":\"2024-04-12T22:36:47.996324498Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodNotScheduled\",\"query\":\"last_over_time(kube_pod_status_unschedulable{namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 1\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} cannot be scheduled for more than 30 minutes.\\nCheck the details of the pod with the following command:\\noc describe -n {{ $labels.namespace }} pod {{ $labels.pod }}\",\"summary\":\"Pod cannot be scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000541001,\"lastEvaluation\":\"2024-04-12T22:36:48.025027584Z\",\"type\":\"alerting\"},{\"name\":\"cluster:alertmanager_integrations:max\",\"query\":\"max(alertmanager_integrations{namespace=\\\"openshift-monitoring\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000121752,\"lastEvaluation\":\"2024-04-12T22:36:47.995154745Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (label_replace(cluster:infra_nodes, \\\"label_node_role_kubernetes_io\\\", \\\"infra\\\", \\\"\\\", \\\"\\\") * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"cpu\\\",unit=\\\"core\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000464569,\"lastEvaluation\":\"2024-04-12T22:36:47.964889868Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_cores_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000130143,\"lastEvaluation\":\"2024-04-12T22:36:47.966423848Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_cpu_sockets_hyperthread_enabled:sum\",\"query\":\"count by (label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (max by (node, package, label_beta_kubernetes_io_instance_type, label_node_hyperthread_enabled, label_node_role_kubernetes_io) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000144733,\"lastEvaluation\":\"2024-04-12T22:36:47.994840189Z\",\"type\":\"recording\"},{\"name\":\"cluster:capacity_memory_bytes:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io) ((cluster:master_nodes * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})) or on (node) (max without (endpoint, instance, job, pod, service) (kube_node_labels) * on (node) group_left () max by (node) (kube_node_status_capacity{resource=\\\"memory\\\",unit=\\\"byte\\\"})))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000365347,\"lastEvaluation\":\"2024-04-12T22:36:47.966555301Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_cpu_usage:ratio\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m])) / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008544355,\"lastEvaluation\":\"2024-04-12T22:36:47.952716206Z\",\"type\":\"recording\"},{\"name\":\"cluster:container_spec_cpu_shares:ratio\",\"query\":\"sum(container_spec_cpu_shares{container=\\\"\\\",pod!=\\\"\\\"}) / 1000 / sum(machine_cpu_cores)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008869612,\"lastEvaluation\":\"2024-04-12T22:36:47.943845054Z\",\"type\":\"recording\"},{\"name\":\"cluster:control_plane:all_nodes_ready\",\"query\":\"sum(min by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}) and max by (node) (kube_node_role{role=\\\"master\\\"})) == bool sum(kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000271916,\"lastEvaluation\":\"2024-04-12T22:36:47.995738736Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_hyperthreading\",\"query\":\"clamp_max(label_replace(sum by (instance, package, core) (node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_replace(label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003e 1, \\\"label_node_hyperthread_enabled\\\", \\\"true\\\", \\\"instance\\\", \\\"(.*)\\\") or on (instance, package) label_replace(sum by (instance, package, core) (label_replace(node_cpu_info{core!=\\\"\\\",package!=\\\"\\\"} or label_join(node_cpu_info{core=\\\"\\\",package=\\\"\\\"}, \\\"core\\\", \\\"\\\", \\\"cpu\\\"), \\\"package\\\", \\\"0\\\", \\\"package\\\", \\\"\\\")) \\u003c= 1, \\\"label_node_hyperthread_enabled\\\", \\\"false\\\", \\\"instance\\\", \\\"(.*)\\\"), 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000775525,\"lastEvaluation\":\"2024-04-12T22:36:47.965356087Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_core_node_labels\",\"query\":\"topk by (node) (1, cluster:nodes_roles) * on (node) group_right (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_node_openshift_io_os_id, label_kubernetes_io_arch, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) label_replace(cluster:cpu_core_hyperthreading, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000288696,\"lastEvaluation\":\"2024-04-12T22:36:47.966133642Z\",\"type\":\"recording\"},{\"name\":\"cluster:cpu_usage_cores:sum\",\"query\":\"sum(1 - rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[2m]) * on (namespace, pod) group_left (node) node_namespace_pod:kube_pod_info:{pod=~\\\"node-exporter.+\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00054354,\"lastEvaluation\":\"2024-04-12T22:36:47.966921808Z\",\"type\":\"recording\"},{\"name\":\"cluster:hyperthread_enabled_nodes\",\"query\":\"kube_node_labels and on (node) (sum by (node, package, core) (label_replace(node_cpu_info, \\\"node\\\", \\\"$1\\\", \\\"instance\\\", \\\"(.*)\\\")) == 2)\",\"labels\":{\"label_node_hyperthread_enabled\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000349047,\"lastEvaluation\":\"2024-04-12T22:36:47.964357608Z\",\"type\":\"recording\"},{\"name\":\"cluster:infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"infra\\\"})\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000170604,\"lastEvaluation\":\"2024-04-12T22:36:47.963852708Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolume_plugin_type_counts:sum\",\"query\":\"sum by (plugin_name, volume_mode) (pv_collector_total_pv_count{volume_plugin!~\\\".*-e2e-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000459179,\"lastEvaluation\":\"2024-04-12T22:36:47.995278127Z\",\"type\":\"recording\"},{\"name\":\"cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_resource_requests_storage_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000755684,\"lastEvaluation\":\"2024-04-12T22:36:47.985956067Z\",\"type\":\"recording\"},{\"name\":\"cluster:kubelet_volume_stats_used_bytes:provisioner:sum\",\"query\":\"sum by (provisioner) (topk by (namespace, persistentvolumeclaim) (1, kubelet_volume_stats_used_bytes) * on (namespace, persistentvolumeclaim) group_right () topk by (namespace, persistentvolumeclaim) (1, kube_persistentvolumeclaim_info * on (storageclass) group_left (provisioner) topk by (storageclass) (1, max by (storageclass, provisioner) (kube_storageclass_info))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000401228,\"lastEvaluation\":\"2024-04-12T22:36:47.987359644Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_infra_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (cluster:master_nodes and on (node) cluster:infra_nodes)\",\"labels\":{\"label_node_role_kubernetes_io_infra\":\"true\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000132182,\"lastEvaluation\":\"2024-04-12T22:36:47.964024441Z\",\"type\":\"recording\"},{\"name\":\"cluster:master_nodes\",\"query\":\"max without (endpoint, instance, job, pod, service) (kube_node_labels and on (node) kube_node_role{role=\\\"master\\\"})\",\"labels\":{\"label_node_role_kubernetes_io\":\"master\",\"label_node_role_kubernetes_io_master\":\"true\",\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000325696,\"lastEvaluation\":\"2024-04-12T22:36:47.963525342Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage:ratio\",\"query\":\"sum by (cluster) (container_memory_usage_bytes{container=\\\"\\\",pod!=\\\"\\\"}) / sum by (cluster) (machine_memory_bytes)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010134309,\"lastEvaluation\":\"2024-04-12T22:36:47.933709305Z\",\"type\":\"recording\"},{\"name\":\"cluster:memory_usage_bytes:sum\",\"query\":\"sum(node_memory_MemTotal_bytes{job=\\\"node-exporter\\\"} - node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000199454,\"lastEvaluation\":\"2024-04-12T22:36:47.967467268Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_instance_type_count:sum\",\"query\":\"sum by (label_beta_kubernetes_io_instance_type, label_node_role_kubernetes_io, label_kubernetes_io_arch, label_node_openshift_io_os_id) (cluster:master_nodes or on (node) kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000256065,\"lastEvaluation\":\"2024-04-12T22:36:47.985698522Z\",\"type\":\"recording\"},{\"name\":\"cluster:nodes_roles\",\"query\":\"cluster:master_infra_nodes or on (node) cluster:master_nodes or on (node) cluster:infra_nodes or on (node) max without (endpoint, instance, job, pod, service) (kube_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000198834,\"lastEvaluation\":\"2024-04-12T22:36:47.964157634Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:containers:sum\",\"query\":\"sum(max by (instance) (kubelet_containers_per_pod_count_sum))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000128573,\"lastEvaluation\":\"2024-04-12T22:36:47.994546103Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_node_ready:avg5m\",\"query\":\"avg_over_time((count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1)) / scalar(count(max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"}))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003232883,\"lastEvaluation\":\"2024-04-12T22:36:48.004967685Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:kube_schedulable_node_ready_reachable:avg5m\",\"query\":\"avg_over_time((((count((max by (node) (up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1) and max by (node) (kube_node_status_condition{condition=\\\"Ready\\\",status=\\\"true\\\"} == 1) and min by (node) (kube_node_spec_unschedulable == 0))) / scalar(count(min by (node) (kube_node_spec_unschedulable == 0))))))[5m:1s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006318392,\"lastEvaluation\":\"2024-04-12T22:36:47.998647403Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:openshift:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace=~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001252884,\"lastEvaluation\":\"2024-04-12T22:36:48.022030176Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:pods:terminal:workload:sum\",\"query\":\"count(count by (namespace, pod) (kube_pod_restart_policy{namespace!~\\\"openshift-.+\\\",type!=\\\"Always\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000127912,\"lastEvaluation\":\"2024-04-12T22:36:47.994417121Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:resources:sum\",\"query\":\"topk(500, max by (resource) (apiserver_storage_objects != -1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002758324,\"lastEvaluation\":\"2024-04-12T22:36:47.991656857Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:max:5m\",\"query\":\"max_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000062601,\"lastEvaluation\":\"2024-04-12T22:36:47.987296113Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:capacity_physical_cpu_cores:min:5m\",\"query\":\"min_over_time(workload:capacity_physical_cpu_cores:sum[5m:15s])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000095062,\"lastEvaluation\":\"2024-04-12T22:36:47.987200051Z\",\"type\":\"recording\"},{\"name\":\"cluster:usage:workload:kube_running_pod_ready:avg\",\"query\":\"avg(kube_running_pod_ready{namespace!~\\\"openshift-.*\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.001740804,\"lastEvaluation\":\"2024-04-12T22:36:48.02328452Z\",\"type\":\"recording\"},{\"name\":\"cluster:virt_platform_nodes:sum\",\"query\":\"count by (type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (sum by (instance, type, system_manufacturer, system_product_name, baseboard_manufacturer, baseboard_product_name) (virt_platform))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000180903,\"lastEvaluation\":\"2024-04-12T22:36:47.964707865Z\",\"type\":\"recording\"},{\"name\":\"instance:etcd_object_counts:sum\",\"query\":\"sum by (instance) (apiserver_storage_objects != -1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.003892765,\"lastEvaluation\":\"2024-04-12T22:36:47.987761702Z\",\"type\":\"recording\"},{\"name\":\"kube_running_pod_ready\",\"query\":\"(max without (condition, container, endpoint, instance, job, service) (((kube_pod_status_ready{condition=\\\"false\\\"} == 1) * 0 or (kube_pod_status_ready{condition=\\\"true\\\"} == 1)) * on (pod, namespace) group_left () group by (pod, namespace) (kube_pod_status_phase{phase=~\\\"Running|Unknown|Pending\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.013824628,\"lastEvaluation\":\"2024-04-12T22:36:48.008202298Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_cpu_usage:sum\",\"query\":\"sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=\\\"\\\",container!=\\\"POD\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.007228534,\"lastEvaluation\":\"2024-04-12T22:36:47.92647816Z\",\"type\":\"recording\"},{\"name\":\"namespace:container_memory_usage_bytes:sum\",\"query\":\"sum by (namespace) (container_memory_usage_bytes{container!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009425742,\"lastEvaluation\":\"2024-04-12T22:36:47.917049358Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_cores:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000163023,\"lastEvaluation\":\"2024-04-12T22:36:47.994675856Z\",\"type\":\"recording\"},{\"name\":\"node_role_os_version_machine:cpu_capacity_sockets:sum\",\"query\":\"count by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (max by (node, package, label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id, label_node_role_kubernetes_io_master, label_node_role_kubernetes_io_infra) (cluster:cpu_core_node_labels))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000167663,\"lastEvaluation\":\"2024-04-12T22:36:47.994985912Z\",\"type\":\"recording\"},{\"name\":\"openshift:cpu_usage_cores:sum\",\"query\":\"cluster:cpu_usage_cores:sum - workload:cpu_usage_cores:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000148032,\"lastEvaluation\":\"2024-04-12T22:36:47.975989784Z\",\"type\":\"recording\"},{\"name\":\"openshift:memory_usage_bytes:sum\",\"query\":\"cluster:memory_usage_bytes:sum - workload:memory_usage_bytes:sum\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000188714,\"lastEvaluation\":\"2024-04-12T22:36:47.985507768Z\",\"type\":\"recording\"},{\"name\":\"pod:container_cpu_usage:sum\",\"query\":\"sum by (pod, namespace) (rate(container_cpu_usage_seconds_total{container=\\\"\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010408544,\"lastEvaluation\":\"2024-04-12T22:36:47.895636169Z\",\"type\":\"recording\"},{\"name\":\"pod:container_fs_usage_bytes:sum\",\"query\":\"sum by (pod, namespace) (container_fs_usage_bytes{pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010997915,\"lastEvaluation\":\"2024-04-12T22:36:47.906047623Z\",\"type\":\"recording\"},{\"name\":\"profile:cluster_monitoring_operator_collection_profile:max\",\"query\":\"max by (profile) (cluster_monitoring_operator_collection_profile == 1)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000066711,\"lastEvaluation\":\"2024-04-12T22:36:47.996011892Z\",\"type\":\"recording\"},{\"name\":\"workload:capacity_physical_cpu_cores:sum\",\"query\":\"(sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_infra=\\\"\\\",label_node_role_kubernetes_io_master=\\\"\\\"} or absent(__does_not_exist__) * 0)) + ((sum(node_role_os_version_machine:cpu_capacity_cores:sum{label_node_role_kubernetes_io_master=\\\"true\\\"} or absent(__does_not_exist__) * 0) * ((max(cluster_master_schedulable == 1) * 0 + 1) or (absent(cluster_master_schedulable == 1) * 0))))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00048553,\"lastEvaluation\":\"2024-04-12T22:36:47.986712971Z\",\"type\":\"recording\"},{\"name\":\"workload:cpu_usage_cores:sum\",\"query\":\"sum(rate(container_cpu_usage_seconds_total{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008318751,\"lastEvaluation\":\"2024-04-12T22:36:47.967667942Z\",\"type\":\"recording\"},{\"name\":\"workload:memory_usage_bytes:sum\",\"query\":\"sum(container_memory_working_set_bytes{container=\\\"\\\",namespace!~\\\"openshift-.+\\\",pod!=\\\"\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009365742,\"lastEvaluation\":\"2024-04-12T22:36:47.976139526Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.130430439,\"lastEvaluation\":\"2024-04-12T22:36:47.895141616Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-monitoring.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"monitoring:container_memory_working_set_bytes:sum\",\"query\":\"sum by (namespace) (max without (instance) (container_memory_working_set_bytes{container=\\\"\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000993363,\"lastEvaluation\":\"2024-04-12T22:36:30.768302406Z\",\"type\":\"recording\"},{\"name\":\"monitoring:haproxy_server_http_responses_total:sum\",\"query\":\"sum by (exported_service) (rate(haproxy_server_http_responses_total{exported_namespace=\\\"openshift-monitoring\\\",exported_service=~\\\"alertmanager-main|prometheus-k8s\\\"}[5m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000488686,\"lastEvaluation\":\"2024-04-12T22:36:30.772285858Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_samples_post_metric_relabeling:topk3\",\"query\":\"topk(3, max by (namespace, job) (topk by (namespace, job) (1, scrape_samples_post_metric_relabeling)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00095385,\"lastEvaluation\":\"2024-04-12T22:36:30.771330297Z\",\"type\":\"recording\"},{\"name\":\"namespace_job:scrape_series_added:topk3_sum1h\",\"query\":\"topk(3, sum by (namespace, job) (sum_over_time(scrape_series_added[1h])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.002031528,\"lastEvaluation\":\"2024-04-12T22:36:30.769297129Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicationController\\\"}, \\\"replicationcontroller\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicationcontroller, namespace) group_left (owner_name) topk by (replicationcontroller, namespace) (1, max by (replicationcontroller, namespace, owner_name) (kube_replicationcontroller_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deploymentconfig\"},\"health\":\"ok\",\"evaluationTime\":0.000470985,\"lastEvaluation\":\"2024-04-12T22:36:30.772775774Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_samples_appended_total:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (rate(prometheus_tsdb_head_samples_appended_total{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[2m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000264694,\"lastEvaluation\":\"2024-04-12T22:36:30.768035902Z\",\"type\":\"recording\"},{\"name\":\"openshift:prometheus_tsdb_head_series:sum\",\"query\":\"sum by (job, namespace) (max without (instance) (prometheus_tsdb_head_series{namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000344929,\"lastEvaluation\":\"2024-04-12T22:36:30.767688283Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.005674618,\"lastEvaluation\":\"2024-04-12T22:36:30.767033763Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-sre.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"code:apiserver_request_total:rate:sum\",\"query\":\"sum by (code) (rate(apiserver_request_total{job=\\\"apiserver\\\"}[10m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.019782155,\"lastEvaluation\":\"2024-04-12T22:36:31.411271461Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.022887027,\"lastEvaluation\":\"2024-04-12T22:36:31.410800849Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"openshift-vsphere.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-3ac9b533-322d-4a25-99e0-914c4dbdc319.yaml\",\"rules\":[{\"name\":\"cluster:vsphere_csi_migration:max\",\"query\":\"max by (status) (vsphere_csi_migration{status=~\\\"|LegacyDeprecatedInTreeDriver|CSIWithMigrationDriver\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000243123,\"lastEvaluation\":\"2024-04-12T22:36:26.80731085Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_esxi_version_total:sum\",\"query\":\"sum by (version) (vsphere_esxi_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000080114,\"lastEvaluation\":\"2024-04-12T22:36:26.807030775Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_infrastructure_failure_domains:max\",\"query\":\"max by (scope) (vsphere_infrastructure_failure_domains)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000066604,\"lastEvaluation\":\"2024-04-12T22:36:26.807243016Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_node_hw_version_total:sum\",\"query\":\"sum by (hw_version) (vsphere_node_hw_version_total)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000065644,\"lastEvaluation\":\"2024-04-12T22:36:26.807112469Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_topology_tags:max\",\"query\":\"max by (source) (vsphere_topology_tags)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000062223,\"lastEvaluation\":\"2024-04-12T22:36:26.807179523Z\",\"type\":\"recording\"},{\"name\":\"cluster:vsphere_vcenter_info:sum\",\"query\":\"sum by (version, build) (vsphere_vcenter_info)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000231123,\"lastEvaluation\":\"2024-04-12T22:36:26.806797263Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000769051,\"lastEvaluation\":\"2024-04-12T22:36:26.806787552Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-state-metrics\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kube-state-metrics-rules-07cb1c2d-024f-4864-ad8b-21ee1306da90.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeStateMetricsListErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_list_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in list operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in list operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000475439,\"lastEvaluation\":\"2024-04-12T22:36:54.944450425Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStateMetricsWatchErrors\",\"query\":\"(sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\",result=\\\"error\\\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_watch_total{job=\\\"kube-state-metrics\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"namespace\":\"openshift-monitoring\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"kube-state-metrics is experiencing errors at an elevated rate in watch operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.\",\"summary\":\"kube-state-metrics is experiencing errors in watch operations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000437549,\"lastEvaluation\":\"2024-04-12T22:36:54.944927434Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001075637,\"lastEvaluation\":\"2024-04-12T22:36:54.943760928Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_cpu_usage_seconds_total\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate\",\"query\":\"sum by (cluster, namespace, pod, container) (irate(container_cpu_usage_seconds_total{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"}[5m])) * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012626512,\"lastEvaluation\":\"2024-04-12T22:36:47.746380067Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.012637473,\"lastEvaluation\":\"2024-04-12T22:36:47.746372897Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_cache\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_cache\",\"query\":\"container_memory_cache{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010943622,\"lastEvaluation\":\"2024-04-12T22:36:35.623483087Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.010261186,\"lastEvaluation\":\"2024-04-12T22:36:35.622071511Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_rss\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_rss\",\"query\":\"container_memory_rss{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.010789669,\"lastEvaluation\":\"2024-04-12T22:36:52.911826657Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.011673021,\"lastEvaluation\":\"2024-04-12T22:36:52.91154551Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_swap\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_swap\",\"query\":\"container_memory_swap{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.011071578,\"lastEvaluation\":\"2024-04-12T22:36:38.910817703Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.010704188,\"lastEvaluation\":\"2024-04-12T22:36:38.910588856Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_memory_working_set_bytes\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_namespace_pod_container:container_memory_working_set_bytes\",\"query\":\"container_memory_working_set_bytes{image!=\\\"\\\",job=\\\"kubelet\\\",metrics_path=\\\"/metrics/cadvisor\\\"} * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=\\\"\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.012711706,\"lastEvaluation\":\"2024-04-12T22:36:50.154884787Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.012724857,\"lastEvaluation\":\"2024-04-12T22:36:50.154876566Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.container_resource\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004433776,\"lastEvaluation\":\"2024-04-12T22:36:42.149576059Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006873023,\"lastEvaluation\":\"2024-04-12T22:36:42.126663768Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_limits\",\"query\":\"kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004633216,\"lastEvaluation\":\"2024-04-12T22:36:42.140182459Z\",\"type\":\"recording\"},{\"name\":\"cluster:namespace:pod_memory:active:kube_pod_container_resource_requests\",\"query\":\"kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"} * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) ((kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.009029006,\"lastEvaluation\":\"2024-04-12T22:36:42.109512855Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005386616,\"lastEvaluation\":\"2024-04-12T22:36:42.154011685Z\",\"type\":\"recording\"},{\"name\":\"namespace_cpu:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.006560217,\"lastEvaluation\":\"2024-04-12T22:36:42.133539481Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_limits:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_limits{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.004755714,\"lastEvaluation\":\"2024-04-12T22:36:42.144817696Z\",\"type\":\"recording\"},{\"name\":\"namespace_memory:kube_pod_container_resource_requests:sum\",\"query\":\"sum by (namespace, cluster) (sum by (namespace, pod, cluster) (max by (namespace, pod, container, cluster) (kube_pod_container_resource_requests{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) * on (namespace, pod, cluster) group_left () max by (namespace, pod, cluster) (kube_pod_status_phase{phase=~\\\"Pending|Running\\\"} == 1)))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.008116488,\"lastEvaluation\":\"2024-04-12T22:36:42.11854467Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.047212386,\"lastEvaluation\":\"2024-04-12T22:36:42.109503685Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"k8s.rules.pod_owner\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"DaemonSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"daemonset\"},\"health\":\"ok\",\"evaluationTime\":0.001090018,\"lastEvaluation\":\"2024-04-12T22:36:38.838936232Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"ReplicaSet\\\"}, \\\"replicaset\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\") * on (replicaset, namespace) group_left (owner_name) topk by (replicaset, namespace) (1, max by (replicaset, namespace, owner_name) (kube_replicaset_owner{job=\\\"kube-state-metrics\\\"})), \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"deployment\"},\"health\":\"ok\",\"evaluationTime\":0.002620491,\"lastEvaluation\":\"2024-04-12T22:36:38.835415538Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"Job\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"job\"},\"health\":\"ok\",\"evaluationTime\":0.000240513,\"lastEvaluation\":\"2024-04-12T22:36:38.840454482Z\",\"type\":\"recording\"},{\"name\":\"namespace_workload_pod:kube_pod_owner:relabel\",\"query\":\"max by (cluster, namespace, workload, pod) (label_replace(kube_pod_owner{job=\\\"kube-state-metrics\\\",owner_kind=\\\"StatefulSet\\\"}, \\\"workload\\\", \\\"$1\\\", \\\"owner_name\\\", \\\"(.*)\\\"))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"workload_type\":\"statefulset\"},\"health\":\"ok\",\"evaluationTime\":0.000424552,\"lastEvaluation\":\"2024-04-12T22:36:38.84002826Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.005396507,\"lastEvaluation\":\"2024-04-12T22:36:38.835301078Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kube-scheduler.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000126387,\"lastEvaluation\":\"2024-04-12T22:36:42.25654512Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000113555,\"lastEvaluation\":\"2024-04-12T22:36:42.255835723Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_binding_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_binding_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000120876,\"lastEvaluation\":\"2024-04-12T22:36:42.255011989Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000096905,\"lastEvaluation\":\"2024-04-12T22:36:42.255950319Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000138198,\"lastEvaluation\":\"2024-04-12T22:36:42.255134295Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_e2e_scheduling_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_e2e_scheduling_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000271785,\"lastEvaluation\":\"2024-04-12T22:36:42.254155403Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000494666,\"lastEvaluation\":\"2024-04-12T22:36:42.256048034Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000559229,\"lastEvaluation\":\"2024-04-12T22:36:42.255274063Z\",\"type\":\"recording\"},{\"name\":\"cluster_quantile:scheduler_scheduling_algorithm_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum without (instance, pod) (rate(scheduler_scheduling_algorithm_duration_seconds_bucket{job=\\\"scheduler\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.000580281,\"lastEvaluation\":\"2024-04-12T22:36:42.254429078Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002529355,\"lastEvaluation\":\"2024-04-12T22:36:42.254145093Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubelet.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.5, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.5\"},\"health\":\"ok\",\"evaluationTime\":0.000744184,\"lastEvaluation\":\"2024-04-12T22:36:46.115656106Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.9, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.9\"},\"health\":\"ok\",\"evaluationTime\":0.000682296,\"lastEvaluation\":\"2024-04-12T22:36:46.114953035Z\",\"type\":\"recording\"},{\"name\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\",\"query\":\"histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m])) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"quantile\":\"0.99\"},\"health\":\"ok\",\"evaluationTime\":0.001092848,\"lastEvaluation\":\"2024-04-12T22:36:46.113858227Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.002420079,\"lastEvaluation\":\"2024-04-12T22:36:46.113845676Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-apps\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"pending\",\"name\":\"KubeContainerWaiting\",\"query\":\"sum by (namespace, pod, container, cluster) (kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"pod/{{ $labels.pod }} in namespace {{ $labels.namespace }} on container {{ $labels.container}} has been in waiting state for longer than 1 hour.\",\"summary\":\"Pod container waiting longer than 1 hour\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubeContainerWaiting\",\"container\":\"node-probe\",\"namespace\":\"openshift-must-gather-dxffw\",\"pod\":\"perf-node-gather-daemonset-b5xc5\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"pod/perf-node-gather-daemonset-b5xc5 in namespace openshift-must-gather-dxffw on container node-probe has been in waiting state for longer than 1 hour.\",\"summary\":\"Pod container waiting longer than 1 hour\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.000662295,\"lastEvaluation\":\"2024-04-12T22:36:28.236250899Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetMisScheduled\",\"query\":\"kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are running where they are not supposed to run.\",\"summary\":\"DaemonSet pods are misscheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000344848,\"lastEvaluation\":\"2024-04-12T22:36:28.237572849Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDaemonSetNotScheduled\",\"query\":\"kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} - kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} are not scheduled.\",\"summary\":\"DaemonSet pods are not scheduled.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000657115,\"lastEvaluation\":\"2024-04-12T22:36:28.236914854Z\",\"type\":\"alerting\"},{\"state\":\"pending\",\"name\":\"KubeDaemonSetRolloutStuck\",\"query\":\"((kube_daemonset_status_current_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_misscheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != 0) or (kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) or (kube_daemonset_status_number_available{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_daemonset_status_desired_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_daemonset_status_updated_number_scheduled{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset }} has not finished or progressed for at least 30 minutes.\",\"summary\":\"DaemonSet rollout is stuck.\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubeDaemonSetRolloutStuck\",\"container\":\"kube-rbac-proxy-main\",\"daemonset\":\"perf-node-gather-daemonset\",\"endpoint\":\"https-main\",\"job\":\"kube-state-metrics\",\"namespace\":\"openshift-must-gather-dxffw\",\"service\":\"kube-state-metrics\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"DaemonSet openshift-must-gather-dxffw/perf-node-gather-daemonset has not finished or progressed for at least 30 minutes.\",\"summary\":\"DaemonSet rollout is stuck.\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"0e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.002392867,\"lastEvaluation\":\"2024-04-12T22:36:28.233856691Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentGenerationMismatch\",\"query\":\"kube_deployment_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_deployment_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment }} does not match, this indicates that the Deployment has failed but has not been rolled back.\",\"summary\":\"Deployment generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001193103,\"lastEvaluation\":\"2024-04-12T22:36:28.228656795Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeDeploymentRolloutStuck\",\"query\":\"kube_deployment_status_condition{condition=\\\"Progressing\\\",job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",status=\\\"false\\\"} != 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Rollout of deployment {{ $labels.namespace }}/{{ $labels.deployment }} is not progressing for longer than 15 minutes.\",\"summary\":\"Deployment rollout is not progressing.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000670776,\"lastEvaluation\":\"2024-04-12T22:36:28.229851058Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaMaxedOut\",\"query\":\"kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has been running at max replicas for longer than 15 minutes.\",\"summary\":\"HPA is running at max replicas\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000459619,\"lastEvaluation\":\"2024-04-12T22:36:28.24056117Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeHpaReplicasMismatch\",\"query\":\"(kube_horizontalpodautoscaler_status_desired_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e kube_horizontalpodautoscaler_spec_min_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003c kube_horizontalpodautoscaler_spec_max_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and changes(kube_horizontalpodautoscaler_status_current_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[15m]) == 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }} has not matched the desired number of replicas for longer than 15 minutes.\",\"summary\":\"HPA has not matched desired number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001479378,\"lastEvaluation\":\"2024-04-12T22:36:28.239080822Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobFailed\",\"query\":\"kube_job_failed{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeJobFailed.md\",\"summary\":\"Job failed to complete.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000314316,\"lastEvaluation\":\"2024-04-12T22:36:28.238765466Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeJobNotCompleted\",\"query\":\"time() - max by (namespace, job_name, cluster) (kube_job_status_start_time{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} and kube_job_status_active{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0) \\u003e 43200\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more than {{ \\\"43200\\\" | humanizeDuration }} to complete.\",\"summary\":\"Job did not complete in time\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000697987,\"lastEvaluation\":\"2024-04-12T22:36:28.237918717Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePodCrashLooping\",\"query\":\"max_over_time(kube_pod_container_status_waiting_reason{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",reason=\\\"CrashLoopBackOff\\\"}[5m]) \\u003e= 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is in waiting state (reason: \\\"CrashLoopBackOff\\\").\",\"summary\":\"Pod is crash looping.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000842105,\"lastEvaluation\":\"2024-04-12T22:36:28.220530281Z\",\"type\":\"alerting\"},{\"state\":\"pending\",\"name\":\"KubePodNotReady\",\"query\":\"sum by (namespace, pod, cluster) (max by (namespace, pod, cluster) (kube_pod_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Pending|Unknown\\\"} unless ignoring (phase) (kube_pod_status_unschedulable{job=\\\"kube-state-metrics\\\"} == 1)) * on (namespace, pod, cluster) group_left (owner_kind) topk by (namespace, pod, cluster) (1, max by (namespace, pod, owner_kind, cluster) (kube_pod_owner{owner_kind!=\\\"Job\\\"}))) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePodNotReady.md\",\"summary\":\"Pod has been in a non-ready state for more than 15 minutes.\"},\"alerts\":[{\"labels\":{\"alertname\":\"KubePodNotReady\",\"namespace\":\"openshift-must-gather-dxffw\",\"pod\":\"perf-node-gather-daemonset-b5xc5\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Pod openshift-must-gather-dxffw/perf-node-gather-daemonset-b5xc5 has been in a non-ready state for longer than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePodNotReady.md\",\"summary\":\"Pod has been in a non-ready state for more than 15 minutes.\"},\"state\":\"pending\",\"activeAt\":\"2024-04-12T22:36:28.219216478Z\",\"value\":\"1e+00\",\"partialResponseStrategy\":\"WARN\"}],\"health\":\"ok\",\"evaluationTime\":0.007222574,\"lastEvaluation\":\"2024-04-12T22:36:28.221432431Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetGenerationMismatch\",\"query\":\"kube_statefulset_status_observed_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_metadata_generation{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset }} does not match, this indicates that the StatefulSet has failed but has not been rolled back.\",\"summary\":\"StatefulSet generation mismatch due to possible roll-back\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000705964,\"lastEvaluation\":\"2024-04-12T22:36:28.231651031Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetReplicasMismatch\",\"query\":\"(kube_statefulset_status_replicas_ready{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[10m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has not matched the expected number of replicas for longer than 15 minutes.\",\"summary\":\"StatefulSet has not matched the expected number of replicas.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001109779,\"lastEvaluation\":\"2024-04-12T22:36:28.230522894Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeStatefulSetUpdateNotRolledOut\",\"query\":\"(max without (revision) (kube_statefulset_status_current_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} unless kube_statefulset_status_update_revision{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) * (kube_statefulset_replicas{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} != kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"})) and (changes(kube_statefulset_status_replicas_updated{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[5m]) == 0)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update has not been rolled out.\",\"summary\":\"StatefulSet update has not been rolled out.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00162914,\"lastEvaluation\":\"2024-04-12T22:36:28.232358075Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.020364443,\"lastEvaluation\":\"2024-04-12T22:36:28.220445469Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-resources\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeCPUOvercommit\",\"query\":\"sum by (cluster) (namespace_cpu:kube_pod_container_resource_requests:sum{job=\\\"kube-state-metrics\\\"}) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"cpu\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted CPU resource requests for Pods by {{ $value }} CPU shares and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted CPU resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558299,\"lastEvaluation\":\"2024-04-12T22:36:38.436962344Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeMemoryOvercommit\",\"query\":\"sum by (cluster) (namespace_memory:kube_pod_container_resource_requests:sum) - (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0 and (sum by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"}) - max by (cluster) (kube_node_status_allocatable{job=\\\"kube-state-metrics\\\",resource=\\\"memory\\\"})) \\u003e 0\",\"duration\":600,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Cluster {{ $labels.cluster }} has overcommitted memory resource requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node failure.\",\"summary\":\"Cluster has overcommitted memory resource requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558559,\"lastEvaluation\":\"2024-04-12T22:36:38.437522304Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaAlmostFull\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) \\u003e 0.9 \\u003c 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota is going to be full.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001061567,\"lastEvaluation\":\"2024-04-12T22:36:38.438082153Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaExceeded\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) \\u003e 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota has exceeded the limits.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000732379,\"lastEvaluation\":\"2024-04-12T22:36:38.439862808Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeQuotaFullyUsed\",\"query\":\"kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"used\\\"} / ignoring (instance, job, type) (kube_resourcequota{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",type=\\\"hard\\\"} \\u003e 0) == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota.\",\"summary\":\"Namespace quota is fully used.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000716728,\"lastEvaluation\":\"2024-04-12T22:36:38.43914473Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.003643934,\"lastEvaluation\":\"2024-04-12T22:36:38.436952983Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-storage\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeErrors\",\"query\":\"kube_persistentvolume_status_phase{job=\\\"kube-state-metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\",phase=~\\\"Failed|Pending\\\"} \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The persistent volume {{ $labels.persistentvolume }} has status {{ $labels.phase }}.\",\"summary\":\"PersistentVolume is having issues with provisioning.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000957141,\"lastEvaluation\":\"2024-04-12T22:36:33.212259431Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeFillingUp\",\"query\":\"(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_capacity_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.03 and kubelet_volume_stats_used_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage }} free.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeFillingUp.md\",\"summary\":\"PersistentVolume is filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001723601,\"lastEvaluation\":\"2024-04-12T22:36:33.206464493Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeFillingUp\",\"query\":\"(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_capacity_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.15 and kubelet_volume_stats_used_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 and predict_linear(kubelet_volume_stats_available_bytes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[6h], 4 * 24 * 3600) \\u003c 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to fill up within four days. Currently {{ $value | humanizePercentage }} is available.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeFillingUp.md\",\"summary\":\"PersistentVolume is filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001571234,\"lastEvaluation\":\"2024-04-12T22:36:33.208189584Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeInodesFillingUp\",\"query\":\"(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_inodes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.03 and kubelet_volume_stats_inodes_used{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":60,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} only has {{ $value | humanizePercentage }} free inodes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeInodesFillingUp.md\",\"summary\":\"PersistentVolumeInodes are filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001091778,\"lastEvaluation\":\"2024-04-12T22:36:33.209761908Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubePersistentVolumeInodesFillingUp\",\"query\":\"(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} / kubelet_volume_stats_inodes{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}) \\u003c 0.15 and kubelet_volume_stats_inodes_used{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} \\u003e 0 and predict_linear(kubelet_volume_stats_inodes_free{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"}[6h], 4 * 24 * 3600) \\u003c 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode=\\\"ReadOnlyMany\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_alerts_k8s_io_kube_persistent_volume_filling_up=\\\"disabled\\\",namespace=~\\\"(openshift-.*|kube-.*|default)\\\"} == 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is expected to run out of inodes within four days. Currently {{ $value | humanizePercentage }} of its inodes are free.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubePersistentVolumeInodesFillingUp.md\",\"summary\":\"PersistentVolumeInodes are filling up.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001403395,\"lastEvaluation\":\"2024-04-12T22:36:33.210854846Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.0067621,\"lastEvaluation\":\"2024-04-12T22:36:33.206456262Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeClientErrors\",\"query\":\"(sum by (cluster, instance, job, namespace) (rate(rest_client_requests_total{code=~\\\"5..\\\",job=\\\"apiserver\\\"}[5m])) / sum by (cluster, instance, job, namespace) (rate(rest_client_requests_total{job=\\\"apiserver\\\"}[5m]))) \\u003e 0.01\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ $value | humanizePercentage }} errors.'\",\"summary\":\"Kubernetes API server client is experiencing errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000975762,\"lastEvaluation\":\"2024-04-12T22:36:46.739292704Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000969879,\"lastEvaluation\":\"2024-04-12T22:36:46.738732156Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system-apiserver\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeAPIDown\",\"query\":\"absent(up{job=\\\"apiserver\\\"} == 1)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"KubeAPI has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeAPIDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000153003,\"lastEvaluation\":\"2024-04-12T22:36:36.883777021Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAPITerminatedRequests\",\"query\":\"sum(rate(apiserver_request_terminations_total{job=\\\"apiserver\\\"}[10m])) / (sum(rate(apiserver_request_total{job=\\\"apiserver\\\"}[10m])) + sum(rate(apiserver_request_terminations_total{job=\\\"apiserver\\\"}[10m]))) \\u003e 0.2\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.\",\"summary\":\"The kubernetes apiserver has terminated {{ $value | humanizePercentage }} of its incoming requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.021009137,\"lastEvaluation\":\"2024-04-12T22:36:36.883945044Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAggregatedAPIDown\",\"query\":\"(1 - max by (name, namespace, cluster) (avg_over_time(aggregator_unavailable_apiservice{job=\\\"apiserver\\\"}[10m]))) * 100 \\u003c 85\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m.\",\"summary\":\"Kubernetes aggregated API is down.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001569211,\"lastEvaluation\":\"2024-04-12T22:36:36.88220667Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeAggregatedAPIErrors\",\"query\":\"sum by (name, namespace, cluster) (increase(aggregator_unavailable_apiservice_total{job=\\\"apiserver\\\"}[10m])) \\u003e 4\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has reported errors. It has appeared unavailable {{ $value | humanize }} times averaged over the past 10m.\",\"summary\":\"Kubernetes aggregated API has reported errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000228844,\"lastEvaluation\":\"2024-04-12T22:36:36.881976606Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.022993415,\"lastEvaluation\":\"2024-04-12T22:36:36.881967456Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"kubernetes-system-kubelet\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"KubeNodeNotReady\",\"query\":\"kube_node_status_condition{condition=\\\"Ready\\\",job=\\\"kube-state-metrics\\\",status=\\\"true\\\"} == 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.node }} has been unready for more than 15 minutes.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeNodeNotReady.md\",\"summary\":\"Node is not ready.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000347466,\"lastEvaluation\":\"2024-04-12T22:36:47.477697163Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeNodeReadinessFlapping\",\"query\":\"sum by (cluster, node) (changes(kube_node_status_condition{condition=\\\"Ready\\\",job=\\\"kube-state-metrics\\\",status=\\\"true\\\"}[15m])) \\u003e 2\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The readiness status of node {{ $labels.node }} has changed {{ $value }} times in the last 15 minutes.\",\"summary\":\"Node readiness status is flapping.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000338697,\"lastEvaluation\":\"2024-04-12T22:36:47.485972463Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeNodeUnreachable\",\"query\":\"(kube_node_spec_taint{effect=\\\"NoSchedule\\\",job=\\\"kube-state-metrics\\\",key=\\\"node.kubernetes.io/unreachable\\\"} unless ignoring (key, value) kube_node_spec_taint{job=\\\"kube-state-metrics\\\",key=~\\\"ToBeDeletedByClusterAutoscaler|cloud.google.com/impending-node-termination|aws-node-termination-handler/spot-itn\\\"}) == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.node }} is unreachable and some workloads may be rescheduled.\",\"summary\":\"Node is unreachable.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000345126,\"lastEvaluation\":\"2024-04-12T22:36:47.47804645Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletClientCertificateRenewalErrors\",\"query\":\"increase(kubelet_certificate_manager_client_expiration_renew_errors[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet on node {{ $labels.node }} has failed to renew its client certificate ({{ $value | humanize }} errors in the last 5 minutes).\",\"summary\":\"Kubelet has failed to renew its client certificate.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000166143,\"lastEvaluation\":\"2024-04-12T22:36:47.489302558Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletDown\",\"query\":\"absent(up{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} == 1)\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Kubelet has disappeared from Prometheus target discovery.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeletDown.md\",\"summary\":\"Target disappeared from Prometheus target discovery.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000211714,\"lastEvaluation\":\"2024-04-12T22:36:47.489627074Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletPlegDurationHigh\",\"query\":\"node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile{quantile=\\\"0.99\\\"} \\u003e= 10\",\"duration\":300,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The Kubelet Pod Lifecycle Event Generator has a 99th percentile duration of {{ $value }} seconds on node {{ $labels.node }}.\",\"summary\":\"Kubelet Pod Lifecycle Event Generator is taking too long to relist.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000133302,\"lastEvaluation\":\"2024-04-12T22:36:47.48631348Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletPodStartUpLatencyHigh\",\"query\":\"histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"}[5m]))) * on (cluster, instance) group_left (node) kubelet_node_name{job=\\\"kubelet\\\",metrics_path=\\\"/metrics\\\"} \\u003e 60\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on node {{ $labels.node }}.\",\"summary\":\"Kubelet Pod startup latency is too high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002851685,\"lastEvaluation\":\"2024-04-12T22:36:47.486448053Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletServerCertificateRenewalErrors\",\"query\":\"increase(kubelet_server_expiration_renew_errors[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Kubelet on node {{ $labels.node }} has failed to renew its server certificate ({{ $value | humanize }} errors in the last 5 minutes).\",\"summary\":\"Kubelet has failed to renew its server certificate.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000155903,\"lastEvaluation\":\"2024-04-12T22:36:47.489469911Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"KubeletTooManyPods\",\"query\":\"count by (cluster, node) ((kube_pod_status_phase{job=\\\"kube-state-metrics\\\",phase=\\\"Running\\\"} == 1) * on (instance, pod, namespace, cluster) group_left (node) topk by (instance, pod, namespace, cluster) (1, kube_pod_info{job=\\\"kube-state-metrics\\\"})) / max by (cluster, node) (kube_node_status_capacity{job=\\\"kube-state-metrics\\\",resource=\\\"pods\\\"} != 1) \\u003e 0.95\",\"duration\":900,\"labels\":{\"namespace\":\"kube-system\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage }} of its Pod capacity.\",\"summary\":\"Kubelet is running at capacity.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.007577107,\"lastEvaluation\":\"2024-04-12T22:36:47.478392976Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.012152905,\"lastEvaluation\":\"2024-04-12T22:36:47.477688673Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kubernetes-monitoring-rules-64ed215d-a46a-4cf2-bc1e-14bb7f48d31c.yaml\",\"rules\":[{\"name\":\":node_memory_MemAvailable_bytes:sum\",\"query\":\"sum by (cluster) (node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"} or (node_memory_Buffers_bytes{job=\\\"node-exporter\\\"} + node_memory_Cached_bytes{job=\\\"node-exporter\\\"} + node_memory_MemFree_bytes{job=\\\"node-exporter\\\"} + node_memory_Slab_bytes{job=\\\"node-exporter\\\"}))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000403148,\"lastEvaluation\":\"2024-04-12T22:36:46.186635932Z\",\"type\":\"recording\"},{\"name\":\"cluster:node_cpu:ratio_rate5m\",\"query\":\"avg by (cluster) (node:node_cpu_utilization:ratio_rate5m)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000116852,\"lastEvaluation\":\"2024-04-12T22:36:46.187712183Z\",\"type\":\"recording\"},{\"name\":\"node:node_cpu_utilization:ratio_rate5m\",\"query\":\"avg by (cluster, node) (sum without (mode) (rate(node_cpu_seconds_total{job=\\\"node-exporter\\\",mode!=\\\"idle\\\",mode!=\\\"iowait\\\",mode!=\\\"steal\\\"}[5m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000670913,\"lastEvaluation\":\"2024-04-12T22:36:46.18704021Z\",\"type\":\"recording\"},{\"name\":\"node_namespace_pod:kube_pod_info:\",\"query\":\"topk by (cluster, namespace, pod) (1, max by (cluster, node, namespace, pod) (label_replace(kube_pod_info{job=\\\"kube-state-metrics\\\",node!=\\\"\\\"}, \\\"pod\\\", \\\"$1\\\", \\\"pod\\\", \\\"(.*)\\\")))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.005129449,\"lastEvaluation\":\"2024-04-12T22:36:46.181502733Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.006339953,\"lastEvaluation\":\"2024-04-12T22:36:46.181491542Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-exporter\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-node-exporter-rules-9b822991-f20a-4cb2-be29-fcb975430f9b.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NodeBondingDegraded\",\"query\":\"(node_bonding_slaves - node_bonding_active) != 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Bonding interface {{ $labels.master }} on {{ $labels.instance }} is in degraded state due to one or more slave failures.\",\"summary\":\"Bonding interface is degraded\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000116593,\"lastEvaluation\":\"2024-04-12T22:36:35.574705821Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeClockNotSynchronising\",\"query\":\"(min_over_time(node_timex_sync_status{job=\\\"node-exporter\\\"}[5m]) == 0 and node_timex_maxerror_seconds{job=\\\"node-exporter\\\"} \\u003e= 16) and on () absent(up{job=\\\"ptp-monitor-service\\\"})\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Clock at {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeClockNotSynchronising.md\",\"summary\":\"Clock not synchronising.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000225155,\"lastEvaluation\":\"2024-04-12T22:36:35.573165571Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeClockSkewDetected\",\"query\":\"((node_timex_offset_seconds{job=\\\"node-exporter\\\"} \\u003e 0.05 and deriv(node_timex_offset_seconds{job=\\\"node-exporter\\\"}[5m]) \\u003e= 0) or (node_timex_offset_seconds{job=\\\"node-exporter\\\"} \\u003c -0.05 and deriv(node_timex_offset_seconds{job=\\\"node-exporter\\\"}[5m]) \\u003c= 0)) and on () absent(up{job=\\\"ptp-monitor-service\\\"})\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Clock at {{ $labels.instance }} is out of sync by more than 0.05s. Ensure NTP is configured correctly on this host.\",\"summary\":\"Clock skew detected.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000332756,\"lastEvaluation\":\"2024-04-12T22:36:35.572831975Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFileDescriptorLimit\",\"query\":\"(node_filefd_allocated{job=\\\"node-exporter\\\"} * 100 / node_filefd_maximum{job=\\\"node-exporter\\\"} \\u003e 90)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"File descriptors limit at {{ $labels.instance }} is currently at {{ printf \\\"%.2f\\\" $value }}%.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFileDescriptorLimit.md\",\"summary\":\"Kernel is predicted to exhaust file descriptors limit soon.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000181364,\"lastEvaluation\":\"2024-04-12T22:36:35.573972417Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFileDescriptorLimit\",\"query\":\"(node_filefd_allocated{job=\\\"node-exporter\\\"} * 100 / node_filefd_maximum{job=\\\"node-exporter\\\"} \\u003e 70)\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"File descriptors limit at {{ $labels.instance }} is currently at {{ printf \\\"%.2f\\\" $value }}%.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFileDescriptorLimit.md\",\"summary\":\"Kernel is predicted to exhaust file descriptors limit soon.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000214924,\"lastEvaluation\":\"2024-04-12T22:36:35.573756543Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfFiles\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 3 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfFiles.md\",\"summary\":\"Filesystem has less than 3% inodes left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00465352,\"lastEvaluation\":\"2024-04-12T22:36:35.566829759Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfFiles\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 5 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfFiles.md\",\"summary\":\"Filesystem has less than 5% inodes left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00460984,\"lastEvaluation\":\"2024-04-12T22:36:35.562218109Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfSpace\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 3 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfSpace.md\",\"summary\":\"Filesystem has less than 3% space left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00414662,\"lastEvaluation\":\"2024-04-12T22:36:35.540116821Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemAlmostOutOfSpace\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 5 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemAlmostOutOfSpace.md\",\"summary\":\"Filesystem has less than 5% space left.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.004205861,\"lastEvaluation\":\"2024-04-12T22:36:35.535909749Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemFilesFillingUp\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 20 and predict_linear(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 4 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left and is filling up fast.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemFilesFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of inodes within the next 4 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.008696209,\"lastEvaluation\":\"2024-04-12T22:36:35.55351983Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemFilesFillingUp\",\"query\":\"(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_files{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 40 and predict_linear(node_filesystem_files_free{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 24 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available inodes left and is filling up.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemFilesFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of inodes within the next 24 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00925279,\"lastEvaluation\":\"2024-04-12T22:36:35.544264851Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemSpaceFillingUp\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 10 and predict_linear(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 4 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left and is filling up fast.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemSpaceFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of space within the next 4 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.008577706,\"lastEvaluation\":\"2024-04-12T22:36:35.527330023Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeFilesystemSpaceFillingUp\",\"query\":\"(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} / node_filesystem_size_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} * 100 \\u003c 15 and predict_linear(node_filesystem_avail_bytes{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"}[6h], 24 * 60 * 60) \\u003c 0 and node_filesystem_readonly{fstype!=\\\"\\\",job=\\\"node-exporter\\\",mountpoint!~\\\"/var/lib/ibmc-s3fs.*\\\"} == 0)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \\\"%.2f\\\" $value }}% available space left and is filling up.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemSpaceFillingUp.md\",\"summary\":\"Filesystem is predicted to run out of space within the next 24 hours.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.010659956,\"lastEvaluation\":\"2024-04-12T22:36:35.516468528Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeHighNumberConntrackEntriesUsed\",\"query\":\"(node_nf_conntrack_entries{job=\\\"node-exporter\\\"} / node_nf_conntrack_entries_limit) \\u003e 0.75\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of conntrack entries are used.\",\"summary\":\"Number of conntrack are getting close to the limit.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000167713,\"lastEvaluation\":\"2024-04-12T22:36:35.572533049Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeMemoryMajorPagesFaults\",\"query\":\"rate(node_vmstat_pgmajfault{job=\\\"node-exporter\\\"}[5m]) \\u003e 500\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Memory major pages are occurring at very high rate at {{ $labels.instance }}, 500 major page faults per second for the last 15 minutes, is currently at {{ printf \\\"%.2f\\\" $value }}.\\nPlease check that there is enough memory available at this instance.\\n\",\"summary\":\"Memory major page faults are occurring at very high rate.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000167984,\"lastEvaluation\":\"2024-04-12T22:36:35.574446406Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeNetworkReceiveErrs\",\"query\":\"rate(node_network_receive_errs_total{job=\\\"node-exporter\\\"}[2m]) / rate(node_network_receive_packets_total{job=\\\"node-exporter\\\"}[2m]) \\u003e 0.01\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \\\"%.0f\\\" $value }} receive errors in the last two minutes.\",\"summary\":\"Network interface is reporting many receive errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000556461,\"lastEvaluation\":\"2024-04-12T22:36:35.571484359Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeNetworkTransmitErrs\",\"query\":\"rate(node_network_transmit_errs_total{job=\\\"node-exporter\\\"}[2m]) / rate(node_network_transmit_packets_total{job=\\\"node-exporter\\\"}[2m]) \\u003e 0.01\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \\\"%.0f\\\" $value }} transmit errors in the last two minutes.\",\"summary\":\"Network interface is reporting many transmit errors.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00049038,\"lastEvaluation\":\"2024-04-12T22:36:35.572041769Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeRAIDDegraded\",\"query\":\"node_md_disks_required{device=~\\\"mmcblk.p.+|nvme.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\\\",job=\\\"node-exporter\\\"} - ignoring (state) (node_md_disks{device=~\\\"mmcblk.p.+|nvme.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\\\",job=\\\"node-exporter\\\",state=\\\"active\\\"}) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"RAID array '{{ $labels.device }}' at {{ $labels.instance }} is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeRAIDDegraded.md\",\"summary\":\"RAID Array is degraded.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000234224,\"lastEvaluation\":\"2024-04-12T22:36:35.573391656Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeRAIDDiskFailure\",\"query\":\"node_md_disks{device=~\\\"mmcblk.p.+|nvme.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\\\",job=\\\"node-exporter\\\",state=\\\"failed\\\"} \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"At least one device in RAID array at {{ $labels.instance }} failed. Array '{{ $labels.device }}' needs attention and possibly a disk swap.\",\"summary\":\"Failed device in RAID array.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000128683,\"lastEvaluation\":\"2024-04-12T22:36:35.57362694Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeSystemSaturation\",\"query\":\"node_load1{job=\\\"node-exporter\\\"} / count without (cpu, mode) (node_cpu_seconds_total{job=\\\"node-exporter\\\",mode=\\\"idle\\\"}) \\u003e 2\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"System load per core at {{ $labels.instance }} has been above 2 for the last 15 minutes, is currently at {{ printf \\\"%.2f\\\" $value }}.\\nThis might indicate this instance resources saturation and can cause it becoming unresponsive.\\n\",\"summary\":\"System saturated, load per core is very high.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000289905,\"lastEvaluation\":\"2024-04-12T22:36:35.574154781Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeSystemdServiceFailed\",\"query\":\"node_systemd_unit_state{job=\\\"node-exporter\\\",state=\\\"failed\\\"} == 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Systemd service {{ $labels.name }} has entered failed state at {{ $labels.instance }}\",\"summary\":\"Systemd service has entered failed state.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000088581,\"lastEvaluation\":\"2024-04-12T22:36:35.57461623Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NodeTextFileCollectorScrapeError\",\"query\":\"node_textfile_scrape_error{job=\\\"node-exporter\\\"} == 1\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Node Exporter text file collector on {{ $labels.instance }} failed to scrape.\",\"summary\":\"Node Exporter text file collector failed to scrape.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000129023,\"lastEvaluation\":\"2024-04-12T22:36:35.572701673Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.057778462,\"lastEvaluation\":\"2024-04-12T22:36:35.516459327Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"node-exporter.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-node-exporter-rules-9b822991-f20a-4cb2-be29-fcb975430f9b.yaml\",\"rules\":[{\"name\":\"instance:node_cpu_utilisation:rate1m\",\"query\":\"1 - avg without (cpu) (sum without (mode) (rate(node_cpu_seconds_total{job=\\\"node-exporter\\\",mode=~\\\"idle|iowait|steal\\\"}[1m])))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.0005563,\"lastEvaluation\":\"2024-04-12T22:36:49.241982597Z\",\"type\":\"recording\"},{\"name\":\"instance:node_load1_per_cpu:ratio\",\"query\":\"(node_load1{job=\\\"node-exporter\\\"} / instance:node_num_cpu:sum{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000207944,\"lastEvaluation\":\"2024-04-12T22:36:49.242540218Z\",\"type\":\"recording\"},{\"name\":\"instance:node_memory_utilisation:ratio\",\"query\":\"1 - ((node_memory_MemAvailable_bytes{job=\\\"node-exporter\\\"} or (node_memory_Buffers_bytes{job=\\\"node-exporter\\\"} + node_memory_Cached_bytes{job=\\\"node-exporter\\\"} + node_memory_MemFree_bytes{job=\\\"node-exporter\\\"} + node_memory_Slab_bytes{job=\\\"node-exporter\\\"})) / node_memory_MemTotal_bytes{job=\\\"node-exporter\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.00046869,\"lastEvaluation\":\"2024-04-12T22:36:49.242749162Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_bytes_excluding_lo:rate1m\",\"query\":\"sum without (device) (rate(node_network_receive_bytes_total{device!=\\\"lo\\\",job=\\\"node-exporter\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000380667,\"lastEvaluation\":\"2024-04-12T22:36:49.243917835Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_receive_drop_excluding_lo:rate1m\",\"query\":\"sum without (device) (rate(node_network_receive_drop_total{device!=\\\"lo\\\",job=\\\"node-exporter\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000468975,\"lastEvaluation\":\"2024-04-12T22:36:49.244581522Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_transmit_bytes_excluding_lo:rate1m\",\"query\":\"sum without (device) (rate(node_network_transmit_bytes_total{device!=\\\"lo\\\",job=\\\"node-exporter\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000266656,\"lastEvaluation\":\"2024-04-12T22:36:49.244300733Z\",\"type\":\"recording\"},{\"name\":\"instance:node_network_transmit_drop_excluding_lo:rate1m\",\"query\":\"sum without (device) (rate(node_network_transmit_drop_total{device!=\\\"lo\\\",job=\\\"node-exporter\\\"}[1m]))\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000495726,\"lastEvaluation\":\"2024-04-12T22:36:49.245052807Z\",\"type\":\"recording\"},{\"name\":\"instance:node_num_cpu:sum\",\"query\":\"count without (cpu, mode) (node_cpu_seconds_total{job=\\\"node-exporter\\\",mode=\\\"idle\\\"})\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000421528,\"lastEvaluation\":\"2024-04-12T22:36:49.241558859Z\",\"type\":\"recording\"},{\"name\":\"instance:node_vmstat_pgmajfault:rate1m\",\"query\":\"rate(node_vmstat_pgmajfault{job=\\\"node-exporter\\\"}[1m])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000145393,\"lastEvaluation\":\"2024-04-12T22:36:49.243219032Z\",\"type\":\"recording\"},{\"name\":\"instance_device:node_disk_io_time_seconds:rate1m\",\"query\":\"rate(node_disk_io_time_seconds_total{device=~\\\"mmcblk.p.+|nvme.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\\\",job=\\\"node-exporter\\\"}[1m])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000265995,\"lastEvaluation\":\"2024-04-12T22:36:49.243366104Z\",\"type\":\"recording\"},{\"name\":\"instance_device:node_disk_io_time_weighted_seconds:rate1m\",\"query\":\"rate(node_disk_io_time_weighted_seconds_total{device=~\\\"mmcblk.p.+|nvme.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\\\",job=\\\"node-exporter\\\"}[1m])\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000281876,\"lastEvaluation\":\"2024-04-12T22:36:49.243634109Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.003523798,\"lastEvaluation\":\"2024-04-12T22:36:49.241547669Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"prometheus\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-prometheus-k8s-prometheus-rules-7874c195-88dc-4d63-a5c5-0194c1fcaa53.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PrometheusBadConfig\",\"query\":\"max_over_time(prometheus_config_last_reload_successful{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has failed to reload its configuration.\",\"summary\":\"Failed Prometheus configuration reload.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000279316,\"lastEvaluation\":\"2024-04-12T22:36:33.887948121Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusDuplicateTimestamps\",\"query\":\"rate(prometheus_target_scrapes_sample_duplicate_timestamp_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} is dropping {{ printf \\\"%.4g\\\" $value  }} samples/s with different values but duplicated timestamp.\",\"summary\":\"Prometheus is dropping samples with duplicate timestamps.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000134973,\"lastEvaluation\":\"2024-04-12T22:36:33.890948168Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusErrorSendingAlertsToSomeAlertmanagers\",\"query\":\"(rate(prometheus_notifications_errors_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) / rate(prometheus_notifications_sent_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m])) * 100 \\u003e 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ printf \\\"%.1f\\\" $value }}% errors while sending alerts from Prometheus {{$labels.namespace}}/{{$labels.pod}} to Alertmanager {{$labels.alertmanager}}.\",\"summary\":\"Prometheus has encountered more than 1% errors sending alerts to a specific Alertmanager.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000307556,\"lastEvaluation\":\"2024-04-12T22:36:33.888631064Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusHighQueryLoad\",\"query\":\"avg_over_time(prometheus_engine_queries{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) / max_over_time(prometheus_engine_queries_concurrent_max{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0.8\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} query API has less than 20% available capacity in its query engine for the last 15 minutes.\",\"summary\":\"Prometheus is reaching its maximum capacity serving concurrent requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000214052,\"lastEvaluation\":\"2024-04-12T22:36:33.895473279Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusLabelLimitHit\",\"query\":\"increase(prometheus_target_scrape_pool_exceeded_label_limits_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has dropped {{ printf \\\"%.0f\\\" $value }} targets because some samples exceeded the configured label_limit, label_name_length_limit or label_value_length_limit.\",\"summary\":\"Prometheus has dropped targets because some scrape configs have exceeded the labels limit.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000127247,\"lastEvaluation\":\"2024-04-12T22:36:33.893930887Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusMissingRuleEvaluations\",\"query\":\"increase(prometheus_rule_group_iterations_missed_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has missed {{ printf \\\"%.0f\\\" $value }} rule group evaluations in the last 5m.\",\"summary\":\"Prometheus is missing rule evaluations due to slow rule group evaluation.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000876046,\"lastEvaluation\":\"2024-04-12T22:36:33.892940045Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusNotConnectedToAlertmanagers\",\"query\":\"max_over_time(prometheus_notifications_alertmanagers_discovered{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003c 1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} is not connected to any Alertmanagers.\",\"summary\":\"Prometheus is not connected to any Alertmanagers.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000145233,\"lastEvaluation\":\"2024-04-12T22:36:33.88893963Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusNotIngestingSamples\",\"query\":\"(rate(prometheus_tsdb_head_samples_appended_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003c= 0 and (sum without (scrape_job) (prometheus_target_metadata_cache_entries{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}) \\u003e 0 or sum without (rule_group) (prometheus_rule_group_rules{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}) \\u003e 0))\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} is not ingesting samples.\",\"summary\":\"Prometheus is not ingesting samples.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00155922,\"lastEvaluation\":\"2024-04-12T22:36:33.889387848Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusNotificationQueueRunningFull\",\"query\":\"(predict_linear(prometheus_notifications_queue_length{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m], 60 * 30) \\u003e min_over_time(prometheus_notifications_queue_capacity{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]))\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Alert notification queue of Prometheus {{$labels.namespace}}/{{$labels.pod}} is running full.\",\"summary\":\"Prometheus alert notification queue predicted to run full in less than 30m.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000263375,\"lastEvaluation\":\"2024-04-12T22:36:33.888366539Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusOutOfOrderTimestamps\",\"query\":\"rate(prometheus_target_scrapes_sample_out_of_order_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} is dropping {{ printf \\\"%.4g\\\" $value  }} samples/s with timestamps arriving out of order.\",\"summary\":\"Prometheus drops samples with out-of-order timestamps.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000131182,\"lastEvaluation\":\"2024-04-12T22:36:33.891084161Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusRemoteStorageFailures\",\"query\":\"((rate(prometheus_remote_storage_failed_samples_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) or rate(prometheus_remote_storage_samples_failed_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m])) / ((rate(prometheus_remote_storage_failed_samples_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) or rate(prometheus_remote_storage_samples_failed_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m])) + (rate(prometheus_remote_storage_succeeded_samples_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) or rate(prometheus_remote_storage_samples_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m])))) * 100 \\u003e 1\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} failed to send {{ printf \\\"%.1f\\\" $value }}% of the samples to {{ $labels.remote_name}}:{{ $labels.url }}\",\"summary\":\"Prometheus fails to send samples to remote storage.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000371897,\"lastEvaluation\":\"2024-04-12T22:36:33.891216323Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusRemoteWriteBehind\",\"query\":\"(max_over_time(prometheus_remote_storage_highest_timestamp_in_seconds{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) - ignoring (remote_name, url) group_right () max_over_time(prometheus_remote_storage_queue_highest_sent_timestamp_seconds{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m])) \\u003e 120\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} remote write is {{ printf \\\"%.1f\\\" $value }}s behind for {{ $labels.remote_name}}:{{ $labels.url }}.\",\"summary\":\"Prometheus remote write is behind.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000222021,\"lastEvaluation\":\"2024-04-12T22:36:33.891604174Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusRemoteWriteDesiredShards\",\"query\":\"(max_over_time(prometheus_remote_storage_shards_desired{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e max_over_time(prometheus_remote_storage_shards_max{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]))\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} remote write desired shards calculation wants to run {{ $value }} shards for queue {{ $labels.remote_name}}:{{ $labels.url }}, which is more than the max of {{ printf `prometheus_remote_storage_shards_max{instance=\\\"%s\\\",job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}` $labels.instance | query | first | value }}.\",\"summary\":\"Prometheus remote write desired shards calculation wants to run more than configured max shards.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00018486,\"lastEvaluation\":\"2024-04-12T22:36:33.891827686Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusRuleFailures\",\"query\":\"increase(prometheus_rule_evaluation_failures_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has failed to evaluate {{ printf \\\"%.0f\\\" $value }} rules in the last 5m.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/PrometheusRuleFailures.md\",\"summary\":\"Prometheus is failing rule evaluations.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000925489,\"lastEvaluation\":\"2024-04-12T22:36:33.892013636Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusSDRefreshFailure\",\"query\":\"increase(prometheus_sd_refresh_failures_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[10m]) \\u003e 0\",\"duration\":1200,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has failed to refresh SD with mechanism {{$labels.mechanism}}.\",\"summary\":\"Failed Prometheus SD refresh.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000136252,\"lastEvaluation\":\"2024-04-12T22:36:33.888229266Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusScrapeBodySizeLimitHit\",\"query\":\"increase(prometheus_target_scrapes_exceeded_body_size_limit_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has failed {{ printf \\\"%.0f\\\" $value }} scrapes in the last 5m because some targets exceeded the configured body_size_limit.\",\"summary\":\"Prometheus has dropped some targets that exceeded body size limit.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000167389,\"lastEvaluation\":\"2024-04-12T22:36:33.894059834Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusScrapeSampleLimitHit\",\"query\":\"increase(prometheus_target_scrapes_exceeded_sample_limit_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has failed {{ printf \\\"%.0f\\\" $value }} scrapes in the last 5m because some targets exceeded the configured sample_limit.\",\"summary\":\"Prometheus has failed scrapes that have exceeded the configured sample limit.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00018596,\"lastEvaluation\":\"2024-04-12T22:36:33.894228303Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusTSDBCompactionsFailing\",\"query\":\"increase(prometheus_tsdb_compactions_failed_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[3h]) \\u003e 0\",\"duration\":14400,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has detected {{$value | humanize}} compaction failures over the last 3h.\",\"summary\":\"Prometheus has issues compacting blocks.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000146163,\"lastEvaluation\":\"2024-04-12T22:36:33.889240606Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusTSDBReloadsFailing\",\"query\":\"increase(prometheus_tsdb_reloads_failures_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[3h]) \\u003e 0\",\"duration\":14400,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has detected {{$value | humanize}} reload failures over the last 3h.\",\"summary\":\"Prometheus has issues reloading blocks from disk.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000153843,\"lastEvaluation\":\"2024-04-12T22:36:33.889085893Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusTargetLimitHit\",\"query\":\"increase(prometheus_target_scrape_pool_exceeded_target_limit_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[5m]) \\u003e 0\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus {{$labels.namespace}}/{{$labels.pod}} has dropped {{ printf \\\"%.0f\\\" $value }} targets because the number of targets exceeded the configured target_limit.\",\"summary\":\"Prometheus has dropped targets because some scrape configs have exceeded the targets limit.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000112886,\"lastEvaluation\":\"2024-04-12T22:36:33.893817161Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusTargetSyncFailure\",\"query\":\"increase(prometheus_target_sync_failed_total{job=~\\\"prometheus-k8s|prometheus-user-workload\\\"}[30m]) \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"{{ printf \\\"%.0f\\\" $value }} targets in Prometheus {{$labels.namespace}}/{{$labels.pod}} have failed to sync because invalid configuration was supplied.\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/PrometheusTargetSyncFailure.md\",\"summary\":\"Prometheus has failed to sync targets.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001056756,\"lastEvaluation\":\"2024-04-12T22:36:33.894415433Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.007430691,\"lastEvaluation\":\"2024-04-12T22:36:33.887940791Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"thanos-sidecar\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-prometheus-k8s-thanos-sidecar-rules-6b89a237-267e-49bd-8a7a-3b4b747aff3b.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ThanosSidecarBucketOperationsFailed\",\"query\":\"sum by (namespace, job, instance) (rate(thanos_objstore_bucket_operation_failures_total{job=~\\\"prometheus-(k8s|user-workload)-thanos-sidecar\\\"}[5m])) \\u003e 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Thanos Sidecar {{$labels.instance}} in {{$labels.namespace}} bucket operations are failing\",\"summary\":\"Thanos Sidecar bucket operations are failing\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.001395957,\"lastEvaluation\":\"2024-04-12T22:36:53.347500223Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ThanosSidecarNoConnectionToStartedPrometheus\",\"query\":\"thanos_sidecar_prometheus_up{job=~\\\"prometheus-(k8s|user-workload)-thanos-sidecar\\\"} == 0 and on (namespace, pod) prometheus_tsdb_data_replay_duration_seconds != 0\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Thanos Sidecar {{$labels.instance}} in {{$labels.namespace}} is unhealthy.\",\"summary\":\"Thanos Sidecar cannot access Prometheus, even though Prometheus seems healthy and has reloaded WAL.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000336737,\"lastEvaluation\":\"2024-04-12T22:36:53.34889912Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001749624,\"lastEvaluation\":\"2024-04-12T22:36:53.347488773Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"config-reloaders\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-prometheus-operator-rules-71ef08d4-8bf3-4307-9073-2f62413e5a75.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ConfigReloaderSidecarErrors\",\"query\":\"max_over_time(reloader_last_reload_successful{namespace=~\\\".+\\\"}[5m]) == 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors encountered while the {{$labels.pod}} config-reloader sidecar attempts to sync config in {{$labels.namespace}} namespace.\\nAs a result, configuration for service running in {{$labels.pod}} may be stale and cannot be updated anymore.\",\"summary\":\"config-reloader sidecar has not had a successful reload for 10m\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.002577597,\"lastEvaluation\":\"2024-04-12T22:36:49.372696769Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.001261024,\"lastEvaluation\":\"2024-04-12T22:36:49.368799956Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"prometheus-operator\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-prometheus-operator-rules-71ef08d4-8bf3-4307-9073-2f62413e5a75.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"PrometheusOperatorListErrors\",\"query\":\"(sum by (cluster, controller, namespace) (rate(prometheus_operator_list_operations_failed_total{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[10m])) / sum by (cluster, controller, namespace) (rate(prometheus_operator_list_operations_total{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[10m]))) \\u003e 0.4\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors while performing List operations in controller {{$labels.controller}} in {{$labels.namespace}} namespace.\",\"summary\":\"Errors while performing list operations in controller.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000558901,\"lastEvaluation\":\"2024-04-12T22:36:47.368434354Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusOperatorNodeLookupErrors\",\"query\":\"rate(prometheus_operator_node_address_lookup_errors_total{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[5m]) \\u003e 0.1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors while reconciling Prometheus in {{ $labels.namespace }} Namespace.\",\"summary\":\"Errors while reconciling Prometheus.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000231415,\"lastEvaluation\":\"2024-04-12T22:36:47.36976782Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusOperatorNotReady\",\"query\":\"min by (cluster, controller, namespace) (max_over_time(prometheus_operator_ready{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[5m]) == 0)\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus operator in {{ $labels.namespace }} namespace isn't ready to reconcile {{ $labels.controller }} resources.\",\"summary\":\"Prometheus operator not ready\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000177293,\"lastEvaluation\":\"2024-04-12T22:36:47.370001155Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusOperatorReconcileErrors\",\"query\":\"(sum by (cluster, controller, namespace) (rate(prometheus_operator_reconcile_errors_total{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[5m]))) / (sum by (cluster, controller, namespace) (rate(prometheus_operator_reconcile_operations_total{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[5m]))) \\u003e 0.1\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"{{ $value | humanizePercentage }} of reconciling operations failed for {{ $labels.controller }} controller in {{ $labels.namespace }} namespace.\",\"summary\":\"Errors while reconciling controller.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000292785,\"lastEvaluation\":\"2024-04-12T22:36:47.369473834Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusOperatorRejectedResources\",\"query\":\"min_over_time(prometheus_operator_managed_resources{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\",state=\\\"rejected\\\"}[5m]) \\u003e 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Prometheus operator in {{ $labels.namespace }} namespace rejected {{ printf \\\"%0.0f\\\" $value }} {{ $labels.controller }}/{{ $labels.resource }} resources.\",\"summary\":\"Resources rejected by Prometheus operator\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000157703,\"lastEvaluation\":\"2024-04-12T22:36:47.370180108Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusOperatorSyncFailed\",\"query\":\"min_over_time(prometheus_operator_syncs{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\",status=\\\"failed\\\"}[5m]) \\u003e 0\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Controller {{ $labels.controller }} in {{ $labels.namespace }} namespace fails to reconcile {{ $value }} objects.\",\"summary\":\"Last controller reconciliation failed\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000154223,\"lastEvaluation\":\"2024-04-12T22:36:47.369318671Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"PrometheusOperatorWatchErrors\",\"query\":\"(sum by (cluster, controller, namespace) (rate(prometheus_operator_watch_operations_failed_total{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[5m])) / sum by (cluster, controller, namespace) (rate(prometheus_operator_watch_operations_total{job=\\\"prometheus-operator\\\",namespace=~\\\"openshift-monitoring|openshift-user-workload-monitoring\\\"}[5m]))) \\u003e 0.4\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Errors while performing watch operations in controller {{$labels.controller}} in {{$labels.namespace}} namespace.\",\"summary\":\"Errors while performing watch operations in controller.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000322487,\"lastEvaluation\":\"2024-04-12T22:36:47.368994925Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.00189359,\"lastEvaluation\":\"2024-04-12T22:36:47.367871569Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"thanos-query\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-thanos-querier-4d6fef75-9643-4de8-b7ab-177750d9fe2f.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"ThanosQueryGrpcClientErrorRate\",\"query\":\"(sum by (namespace, job) (rate(grpc_client_handled_total{grpc_code!=\\\"OK\\\",job=\\\"thanos-querier\\\"}[5m])) / sum by (namespace, job) (rate(grpc_client_started_total{job=\\\"thanos-querier\\\"}[5m]))) * 100 \\u003e 5\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Thanos Query {{$labels.job}} in {{$labels.namespace}} is failing to send {{$value | humanize}}% of requests.\",\"summary\":\"Thanos Query is failing to send requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000228403,\"lastEvaluation\":\"2024-04-12T22:36:37.974581115Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ThanosQueryGrpcServerErrorRate\",\"query\":\"(sum by (namespace, job) (rate(grpc_server_handled_total{grpc_code=~\\\"Unknown|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\\\",job=\\\"thanos-querier\\\"}[5m])) / sum by (namespace, job) (rate(grpc_server_started_total{job=\\\"thanos-querier\\\"}[5m])) * 100 \\u003e 5)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Thanos Query {{$labels.job}} in {{$labels.namespace}} is failing to handle {{$value | humanize}}% of requests.\",\"summary\":\"Thanos Query is failing to handle requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000960842,\"lastEvaluation\":\"2024-04-12T22:36:37.973619373Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ThanosQueryHighDNSFailures\",\"query\":\"(sum by (namespace, job) (rate(thanos_query_store_apis_dns_failures_total{job=\\\"thanos-querier\\\"}[5m])) / sum by (namespace, job) (rate(thanos_query_store_apis_dns_lookups_total{job=\\\"thanos-querier\\\"}[5m]))) * 100 \\u003e 1\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Thanos Query {{$labels.job}} in {{$labels.namespace}} have {{$value | humanize}}% of failing DNS queries for store endpoints.\",\"summary\":\"Thanos Query is having high number of DNS failures.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000226972,\"lastEvaluation\":\"2024-04-12T22:36:37.974810897Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ThanosQueryHttpRequestQueryErrorRateHigh\",\"query\":\"(sum by (namespace, job) (rate(http_requests_total{code=~\\\"5..\\\",handler=\\\"query\\\",job=\\\"thanos-querier\\\"}[5m])) / sum by (namespace, job) (rate(http_requests_total{handler=\\\"query\\\",job=\\\"thanos-querier\\\"}[5m]))) * 100 \\u003e 5\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Thanos Query {{$labels.job}} in {{$labels.namespace}} is failing to handle {{$value | humanize}}% of \\\"query\\\" requests.\",\"summary\":\"Thanos Query is failing to handle requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000477135,\"lastEvaluation\":\"2024-04-12T22:36:37.972935817Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ThanosQueryHttpRequestQueryRangeErrorRateHigh\",\"query\":\"(sum by (namespace, job) (rate(http_requests_total{code=~\\\"5..\\\",handler=\\\"query_range\\\",job=\\\"thanos-querier\\\"}[5m])) / sum by (namespace, job) (rate(http_requests_total{handler=\\\"query_range\\\",job=\\\"thanos-querier\\\"}[5m]))) * 100 \\u003e 5\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Thanos Query {{$labels.job}} in {{$labels.namespace}} is failing to handle {{$value | humanize}}% of \\\"query_range\\\" requests.\",\"summary\":\"Thanos Query is failing to handle requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.00020264,\"lastEvaluation\":\"2024-04-12T22:36:37.973415023Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"ThanosQueryOverload\",\"query\":\"(max_over_time(thanos_query_concurrent_gate_queries_max[5m]) - avg_over_time(thanos_query_concurrent_gate_queries_in_flight[5m]) \\u003c 1)\",\"duration\":3600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Thanos Query {{$labels.job}} in {{$labels.namespace}} has been overloaded for more than 15 minutes. This may be a symptom of excessive simultanous complex requests, low performance of the Prometheus API, or failures within these components. Assess the health of the Thanos query instances, the connnected Prometheus instances, look for potential senders of these requests and then contact support.\",\"summary\":\"Thanos query reaches its maximum capacity serving concurrent requests.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000175909,\"lastEvaluation\":\"2024-04-12T22:36:37.975039089Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.002288402,\"lastEvaluation\":\"2024-04-12T22:36:37.972928707Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"multus-admission-controller-monitor-service.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-multus-prometheus-k8s-rules-395f7708-c8ad-4dce-9fed-7f020ab87190.yaml\",\"rules\":[{\"name\":\"cluster:network_attachment_definition_enabled_instance_up:max\",\"query\":\"max by (networks) (network_attachment_definition_enabled_instance_up)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000309546,\"lastEvaluation\":\"2024-04-12T22:36:36.830715752Z\",\"type\":\"recording\"},{\"name\":\"cluster:network_attachment_definition_instances:max\",\"query\":\"max by (networks) (network_attachment_definition_instances)\",\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\"},\"health\":\"ok\",\"evaluationTime\":0.000805556,\"lastEvaluation\":\"2024-04-12T22:36:36.831027648Z\",\"type\":\"recording\"}],\"interval\":30,\"evaluationTime\":0.000970292,\"lastEvaluation\":\"2024-04-12T22:36:36.830408913Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"olm.csv_abnormal.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-operator-lifecycle-manager-olm-alert-rules-4bf31be1-9a2c-4ae7-8f24-8ab637657ab8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"CsvAbnormalFailedOver2Min\",\"query\":\"last_over_time(csv_abnormal{phase=\\\"Failed\\\"}[5m])\",\"duration\":120,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failed to install Operator {{ $labels.name }} version {{ $labels.version }}. Reason-{{ $labels.reason }}\",\"summary\":\"CSV failed for over 2 minutes\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000217684,\"lastEvaluation\":\"2024-04-12T22:36:47.6266057Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"CsvAbnormalOver30Min\",\"query\":\"last_over_time(csv_abnormal{phase=~\\\"(Replacing|Pending|Deleting|Unknown)\\\"}[5m])\",\"duration\":1800,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"Failed to install Operator {{ $labels.name }} version {{ $labels.version }}. Phase-{{ $labels.phase }} Reason-{{ $labels.reason }}\",\"summary\":\"CSV abnormal for over 30 minutes\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000159713,\"lastEvaluation\":\"2024-04-12T22:36:47.626825914Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000304236,\"lastEvaluation\":\"2024-04-12T22:36:47.625855665Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"olm.installplan.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-operator-lifecycle-manager-olm-alert-rules-4bf31be1-9a2c-4ae7-8f24-8ab637657ab8.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"InstallPlanStepAppliedWithWarnings\",\"query\":\"sum by (namespace) (increase(installplan_warnings_total[5m])) \\u003e 0\",\"duration\":0,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"The API server returned a warning during installation or upgrade of an operator. An Event with reason \\\"AppliedWithWarnings\\\" has been created with complete details, including a reference to the InstallPlan step that generated the warning.\",\"summary\":\"API returned a warning when modifying an operator\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000310166,\"lastEvaluation\":\"2024-04-12T22:36:35.372140705Z\",\"type\":\"alerting\"}],\"interval\":30,\"evaluationTime\":0.000322637,\"lastEvaluation\":\"2024-04-12T22:36:35.372132065Z\",\"limit\":0,\"partialResponseStrategy\":\"ABORT\"},{\"name\":\"cluster-network-operator-master.rules\",\"file\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-ovn-kubernetes-master-rules-4ccca175-5dbc-4f7b-9ef7-56fb48aa2d83.yaml\",\"rules\":[{\"state\":\"inactive\",\"name\":\"NoOvnClusterManagerLeader\",\"query\":\"max by (namespace) (max_over_time(ovnkube_clustermanager_leader[5m])) == 0\",\"duration\":300,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Networking control plane is degraded. Networking configuration updates applied to the cluster will not be\\nimplemented while there is no OVN Kubernetes leader. Existing workloads should continue to have connectivity.\\nOVN-Kubernetes control plane is not functional.\\n\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-network-operator/NoOvnMasterLeader.md\",\"summary\":\"There is no ovn-kubernetes cluster manager leader.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000174633,\"lastEvaluation\":\"2024-04-12T22:36:53.955823277Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NoRunningOvnControlPlane\",\"query\":\"absent(up{job=\\\"ovnkube-control-plane\\\",namespace=\\\"openshift-ovn-kubernetes\\\"} == 1)\",\"duration\":300,\"labels\":{\"namespace\":\"openshift-ovn-kubernetes\",\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Networking control plane is degraded. Networking configuration updates applied to the cluster will not be\\nimplemented while there are no OVN Kubernetes pods.\\n\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-network-operator/NoRunningOvnMaster.md\",\"summary\":\"There is no running ovn-kubernetes control plane.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000182333,\"lastEvaluation\":\"2024-04-12T22:36:53.955639404Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"NorthboundStale\",\"query\":\"time() - max_over_time(ovnkube_controller_nb_e2e_timestamp[5m]) \\u003e 120\",\"duration\":600,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Networking control plane is degraded. Networking configuration updates applied to the cluster will not be\\nimplemented. Existing workloads should continue to have connectivity. OVN-Kubernetes control plane and/or\\nOVN northbound database may not be functional.\\n\",\"runbook_url\":\"https://github.com/openshift/runbooks/blob/master/alerts/cluster-network-operator/NorthboundStaleAlert.md\",\"summary\":\"ovn-kubernetes has not written anything to the northbound database for too long.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000213554,\"lastEvaluation\":\"2024-04-12T22:36:53.95599925Z\",\"type\":\"alerting\"},{\"state\":\"inactive\",\"name\":\"OVNKubernetesNorthboundDatabaseCPUUsageHigh\",\"query\":\"(sum by (instance, name, namespace) (rate(container_cpu_usage_seconds_total{container=\\\"nbdb\\\"}[5m]))) \\u003e 0.8\",\"duration\":900,\"labels\":{\"prometheus\":\"openshift-monitoring/k8s\",\"severity\":\"info\"},\"annotations\":{\"description\":\"High OVN northbound CPU usage indicates high load on the networking control plane.\",\"summary\":\"OVN northbound database {{ $labels.instance }} is greater than {{ $value | humanizePercentage }} percent CPU usage for a period of time.\"},\"alerts\":[],\"health\":\"ok\",\"evaluationTime\":0.000338837
Ginkgo exit error 1: exit with code 1