May 14 14:58:03.685 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-177-47.us-west-2.compute.internal uid/c71f8531-102b-4d48-b934-9462b47adb95 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-14T14:34:33.4719601Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=rhaos-4.12-rhel-8, revision=a2a6d69)"\nlevel=info ts=2024-05-14T14:34:33.472025855Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20240509-20:14:50)"\nlevel=info ts=2024-05-14T14:34:33.472194287Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-14T14:34:37.306675388Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-14T14:34:37.306769795Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-14T14:34:38.478265069Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-14T14:36:18.011644697Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-14T14:40:07.032479808Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefile
May 14 14:58:20.555 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.57: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-18720219e4d6e2aee48121a79f291db9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2cdc7cfafc8b803dc63c2fe25b1be4f7, retrying]
May 14 14:58:20.555 - 522s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.57: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-18720219e4d6e2aee48121a79f291db9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2cdc7cfafc8b803dc63c2fe25b1be4f7, retrying]

... 1 lines not shown

May 07 14:59:25.309 E ns/e2e-k8s-sig-apps-job-upgrade-7394 pod/foo-hqbsk node/ip-10-0-190-212.us-west-2.compute.internal uid/9d3ef93b-1249-4c38-b57a-9af5f36236a8 container/c reason/ContainerExit code/137 cause/Error
May 07 15:01:22.476 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.56: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cfbdf081b58b8712879802eca43ba650 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-84bef60453bda3335574fd165c730e0c, retrying]
May 07 15:01:22.476 - 485s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.56: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cfbdf081b58b8712879802eca43ba650 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-84bef60453bda3335574fd165c730e0c, retrying]

... 1 lines not shown