May 16 23:08:47.963 E ns/e2e-k8s-sig-apps-job-upgrade-2695 pod/foo-thm6d node/ip-10-0-158-47.us-west-2.compute.internal uid/2def5905-a781-4b90-a9b5-0eda5448ae29 container/c reason/ContainerExit code/137 cause/Error
May 16 23:10:20.229 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-16-144839: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-94314601f203a3429f900bd0b1b4d8d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-bc6d8f7e94a2a73d80d07dc61c8f94a3, retrying]
May 16 23:10:20.229 - 535s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-16-144839: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-94314601f203a3429f900bd0b1b4d8d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-bc6d8f7e94a2a73d80d07dc61c8f94a3, retrying]

... 1 lines not shown

May 15 19:34:32.902 - 80ms  E clusteroperator/authentication condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
May 15 19:35:18.475 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-15-175025: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f64b75f37343fab002f7f473297827ec expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-95313b9fe00f522fcec9c0b6881919ba, retrying]
May 15 19:35:18.475 - 526s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-15-175025: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f64b75f37343fab002f7f473297827ec expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-95313b9fe00f522fcec9c0b6881919ba, retrying]

... 1 lines not shown

May 16 16:34:28.060 E ns/openshift-multus pod/multus-additional-cni-plugins-5cxfn node/ip-10-0-141-7.ec2.internal uid/3ea5bf9b-afd6-4084-bbc1-858cff9327f8 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 16:34:38.338 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-16-144839: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fd3b9023b89b47e4eceac4951cf7a5c9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-9769aa5373c1ad0035fa05106ae5ad9c, retrying]
May 16 16:34:38.338 - 482s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-16-144839: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fd3b9023b89b47e4eceac4951cf7a5c9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-9769aa5373c1ad0035fa05106ae5ad9c, retrying]

... 1 lines not shown

May 15 11:07:26.573 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-167-10.us-west-1.compute.internal uid/ffb7906b-36e1-46fc-bc45-f89d7de60eed container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-15T11:03:34.576761468Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=rhaos-4.12-rhel-8, revision=f5b8e09)"\nlevel=info ts=2024-05-15T11:03:34.576823859Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20240514-17:35:17)"\nlevel=info ts=2024-05-15T11:03:34.576970941Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-15T11:03:40.708121124Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-15T11:03:40.708214695Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\n
May 15 11:07:58.832 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-15-091401: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8e2cca9bdbab9931f80539073c88be73 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-283840418906fc95358d148c2f188cad, retrying]
May 15 11:07:58.832 - 140s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-15-091401: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8e2cca9bdbab9931f80539073c88be73 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-283840418906fc95358d148c2f188cad, retrying]

... 1 lines not shown

May 15 03:30:06.098 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-166-185.us-west-1.compute.internal uid/e0a6cb06-2164-4cd1-b2fd-4fb9688ba061 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-15T03:30:02.100Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=85c9b46fcf2890772ad3c7f7bdbeaadb7a1f5273)"\nts=2024-05-15T03:30:02.100Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@a542505153bf, date=20240514-16:37:45)"\nts=2024-05-15T03:30:02.119Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-15T03:30:02.154Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-15T03:30:02.154Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-15T03:30:02.154Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=34.252663ms\n
May 15 03:30:36.439 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-15-013755: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e5ca72dc2a21995bb839d4dce1cfd5c8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-53c817afaf9c88613a7a6b6baa70a583, retrying]
May 15 03:30:36.439 - 143s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-15-013755: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e5ca72dc2a21995bb839d4dce1cfd5c8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-53c817afaf9c88613a7a6b6baa70a583, retrying]

... 1 lines not shown

May 15 17:43:13.345 E ns/e2e-k8s-sig-apps-job-upgrade-3295 pod/foo-m96sz node/ip-10-0-161-150.us-west-1.compute.internal uid/6d08f854-3913-474f-a4ed-607d163a9a2b container/c reason/ContainerExit code/137 cause/Error
May 15 17:44:07.872 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-15-160118: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f8deaee7b9cbc00606ccbb7f632ba2e2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-03fe01019d107ab82b4e693a96b27519, retrying]
May 15 17:44:07.872 - 527s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-15-160118: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f8deaee7b9cbc00606ccbb7f632ba2e2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-03fe01019d107ab82b4e693a96b27519, retrying]

... 1 lines not shown

May 15 01:34:47.441 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-131-142.us-west-2.compute.internal uid/f6c0dab5-c16d-41e7-9c33-34d31af24135 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-15T01:34:44.543Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=85c9b46fcf2890772ad3c7f7bdbeaadb7a1f5273)"\nts=2024-05-15T01:34:44.543Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@a542505153bf, date=20240514-16:37:45)"\nts=2024-05-15T01:34:44.585Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-15T01:34:44.618Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-15T01:34:44.618Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-15T01:34:44.618Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=32.566445ms\n
May 15 01:35:20.207 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-14-234113: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3505e34cbd9cb35d8f51d8587a1b2058 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fb094bcc46696978b2ae7780e9c54714, retrying]
May 15 01:35:20.207 - 135s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-14-234113: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3505e34cbd9cb35d8f51d8587a1b2058 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fb094bcc46696978b2ae7780e9c54714, retrying]

... 1 lines not shown

May 14 15:04:59.499 E ns/e2e-k8s-sig-apps-job-upgrade-7415 pod/foo-zt2p7 node/ip-10-0-147-141.us-west-2.compute.internal uid/92da7df1-73a0-4c7d-9377-4ac148c9c6cc container/c reason/ContainerExit code/137 cause/Error
May 14 15:06:12.267 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-14-132145: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f299fff70c0b4b586c15327e388d9dae expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-62dca74d729a4074b4017ff9399dd1f0, retrying]
May 14 15:06:12.267 - 527s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-14-132145: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f299fff70c0b4b586c15327e388d9dae expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-62dca74d729a4074b4017ff9399dd1f0, retrying]

... 1 lines not shown

May 14 23:27:56.646 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-144-148.ec2.internal uid/1b731d46-b9c1-4946-b353-39ce5b2d153c container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-14T23:27:54.702Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=85c9b46fcf2890772ad3c7f7bdbeaadb7a1f5273)"\nts=2024-05-14T23:27:54.702Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@4cb74412faf8, date=20240514-16:39:22)"\nts=2024-05-14T23:27:54.736Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-14T23:27:54.787Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-14T23:27:54.787Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-14T23:27:54.787Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=50.689124ms\n
May 14 23:28:08.164 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-14-213435: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ad8e78aabc956c0245fc3f224829fb4d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-96def37d0ec1889144ba7f860f4108d9, retrying]
May 14 23:28:08.164 - 516s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-14-213435: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ad8e78aabc956c0245fc3f224829fb4d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-96def37d0ec1889144ba7f860f4108d9, retrying]

... 1 lines not shown

May 14 00:23:10.183 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-132-255.us-west-2.compute.internal uid/de1a9d14-9139-4013-b1ee-2ca81903456b container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-14T00:23:06.539Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=49098c3442a3073e00e8564e75900cdffe96e785)"\nts=2024-05-14T00:23:06.539Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@96e9cf12ccaa, date=20240509-19:58:48)"\nts=2024-05-14T00:23:06.562Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-14T00:23:06.590Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-14T00:23:06.590Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-14T00:23:06.590Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=28.212192ms\n
May 14 00:23:18.785 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-13-223737: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c927af6f7b2ada23500e38cd5355d8a5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4bbb0acf219a804eac06190c3f6c2c12, retrying]
May 14 00:23:18.785 - 147s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-13-223737: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c927af6f7b2ada23500e38cd5355d8a5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4bbb0acf219a804eac06190c3f6c2c12, retrying]

... 1 lines not shown

May 13 22:57:01.974 E ns/e2e-k8s-sig-apps-job-upgrade-6877 pod/foo-v7rfv node/ip-10-0-219-59.us-west-2.compute.internal uid/f77e4951-72ee-4680-abc6-0733ed2b095e container/c reason/ContainerExit code/137 cause/Error
May 13 22:59:00.278 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-11-115309: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8b42bdef34c385888a8976f62fe95efb expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6522ddd9e925da71eb255a82d6a428b7, retrying]
May 13 22:59:00.278 - 124s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-11-115309: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8b42bdef34c385888a8976f62fe95efb expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6522ddd9e925da71eb255a82d6a428b7, retrying]

... 1 lines not shown

May 11 13:35:13.416 E ns/openshift-machine-config-operator pod/machine-config-daemon-2vbg8 node/ip-10-0-225-175.us-east-2.compute.internal uid/aaee0256-4b57-4ad6-ba12-84c4489caaf1 container/oauth-proxy reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 11 13:35:14.167 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-11-115309: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-954b11c7ba0678b46d33a7e137254869 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6b850ceaaf4c6e9716abf4ad45d97e56, retrying]
May 11 13:35:14.167 - 103s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-11-115309: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-954b11c7ba0678b46d33a7e137254869 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6b850ceaaf4c6e9716abf4ad45d97e56, retrying]

... 1 lines not shown

May 11 03:41:53.562 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-219-98.ec2.internal uid/30709c2d-0a29-41b2-a8a9-c9ee781bdc2a container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-11T03:41:50.606Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=49098c3442a3073e00e8564e75900cdffe96e785)"\nts=2024-05-11T03:41:50.606Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@96e9cf12ccaa, date=20240509-19:58:48)"\nts=2024-05-11T03:41:50.643Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-11T03:41:50.667Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-11T03:41:50.667Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-11T03:41:50.667Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=24.263519ms\n
May 11 03:44:00.961 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-11-020052: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fac8ab05b00207acc043b938c394d99f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a7dcab65517326b08220aa3ab39367b4, retrying]
May 11 03:44:00.961 - 507s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-11-020052: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fac8ab05b00207acc043b938c394d99f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a7dcab65517326b08220aa3ab39367b4, retrying]

... 1 lines not shown

May 11 05:04:15.000 - 5s    E ns/openshift-image-registry route/test-disruption-new disruption/image-registry connection/new reason/DisruptionBegan ns/openshift-image-registry route/test-disruption-new disruption/image-registry connection/new stopped responding to GET requests over new connections: Get "https://test-disruption-new-openshift-image-registry.apps.ci-op-yby16f8w-c43d5.aws-2.ci.openshift.org/healthz": net/http: TLS handshake timeout
May 11 05:04:22.622 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-11-030728: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14f796318beb76b8caf44187cdeaf016 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5b115e6dfb805889a6251c3a6af8627e, retrying]
May 11 05:04:22.622 - 162s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-11-030728: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14f796318beb76b8caf44187cdeaf016 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5b115e6dfb805889a6251c3a6af8627e, retrying]

... 1 lines not shown

May 10 15:46:30.851 E ns/e2e-k8s-sig-apps-job-upgrade-5666 pod/foo-mps4r node/ip-10-0-255-123.us-east-2.compute.internal uid/cd104c5f-31d9-4086-b05b-eb6876ccb731 container/c reason/ContainerExit code/137 cause/Error
May 10 15:48:18.440 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-10-140820: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ff883fcdb27a058141da9a900d9075a9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-12d2e1512ba112d3121272752a95c4e4, retrying]
May 10 15:48:18.440 - 491s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-10-140820: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ff883fcdb27a058141da9a900d9075a9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-12d2e1512ba112d3121272752a95c4e4, retrying]

... 1 lines not shown

May 10 07:07:54.808 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-149-42.ec2.internal uid/dd822729-8dc6-444f-8386-e7e3a5c7a8a2 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-10T06:37:24.260501283Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=rhaos-4.12-rhel-8, revision=a2a6d69)"\nlevel=info ts=2024-05-10T06:37:24.260566694Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20240509-20:14:50)"\nlevel=info ts=2024-05-10T06:37:24.260731767Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-10T06:37:29.087683711Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T06:37:29.087777133Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T06:37:31.887876951Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T06:38:57.227323753Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T06:40:26.223895746Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefi
May 10 07:08:45.148 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-10-051645: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9bd53e206472b2935211f3de1652fcbe expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ff6496dabdd7895610fd2207236fbed7, retrying]
May 10 07:08:45.148 - 155s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-10-051645: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9bd53e206472b2935211f3de1652fcbe expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ff6496dabdd7895610fd2207236fbed7, retrying]

... 1 lines not shown

May 10 05:58:12.574 E ns/e2e-k8s-sig-apps-job-upgrade-2497 pod/foo-6ncqf node/ip-10-0-224-236.us-east-2.compute.internal uid/cdf89f25-ca3b-4f28-bf84-cfbd7c86a99f container/c reason/ContainerExit code/137 cause/Error
May 10 05:58:49.672 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-10-041745: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b446fab61c0e651a55ecc9a8b0d7cb00 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-9c6a1c249bd9a13f3ba4c2cf18fb9c60, retrying]
May 10 05:58:49.672 - 512s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-10-041745: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b446fab61c0e651a55ecc9a8b0d7cb00 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-9c6a1c249bd9a13f3ba4c2cf18fb9c60, retrying]

... 1 lines not shown

May 10 05:03:01.164 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-136-153.us-west-1.compute.internal uid/655a0f00-6040-42a8-aa18-8e443d13ce96 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-10T05:02:58.221Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=49098c3442a3073e00e8564e75900cdffe96e785)"\nts=2024-05-10T05:02:58.221Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@164b64d0a0d4, date=20240509-19:59:44)"\nts=2024-05-10T05:02:58.245Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-10T05:02:58.279Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-10T05:02:58.279Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-10T05:02:58.279Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=33.698374ms\n
May 10 05:03:05.596 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-10-031059: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5821259e78a4b18d9c587b09f85e0265 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b760dc691627f0ea77591a6cb5cb2749, retrying]
May 10 05:03:05.596 - 151s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-10-031059: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5821259e78a4b18d9c587b09f85e0265 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b760dc691627f0ea77591a6cb5cb2749, retrying]

... 1 lines not shown

May 10 04:03:58.710 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-157-165.us-west-2.compute.internal uid/dd31d32a-55bf-4b7a-990a-d3b618fb7a36 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-10T04:03:50.955Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=49098c3442a3073e00e8564e75900cdffe96e785)"\nts=2024-05-10T04:03:50.955Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@96e9cf12ccaa, date=20240509-19:58:48)"\nts=2024-05-10T04:03:51.009Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-10T04:03:51.040Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-10T04:03:51.040Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-10T04:03:51.040Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=31.672272ms\n
May 10 04:04:22.961 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-10-021917: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-51665ad5c25dc4f216d62673f43f77b0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-571a21167cf1a82f11058b5c0065d8f2, retrying]
May 10 04:04:22.961 - 133s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-10-021917: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-51665ad5c25dc4f216d62673f43f77b0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-571a21167cf1a82f11058b5c0065d8f2, retrying]

... 1 lines not shown

May 10 03:42:54.484 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-138-115.us-west-1.compute.internal uid/b3fb4935-7e7a-4a51-8b82-60347ce9f6c0 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-10T03:42:50.177Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=49098c3442a3073e00e8564e75900cdffe96e785)"\nts=2024-05-10T03:42:50.177Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@96e9cf12ccaa, date=20240509-19:58:48)"\nts=2024-05-10T03:42:50.193Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-10T03:42:50.230Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-10T03:42:50.230Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-10T03:42:50.230Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=37.077199ms\n
May 10 03:43:48.722 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-10-015909: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-227878888397f670f04b0df7c8212c4e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-07c96a0aa27c572f18f3f5334b77d011, retrying]
May 10 03:43:48.722 - 130s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-10-015909: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-227878888397f670f04b0df7c8212c4e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-07c96a0aa27c572f18f3f5334b77d011, retrying]

... 1 lines not shown

May 10 01:14:29.374 - 35s   E clusteroperator/network condition/Degraded status/True reason/Error while updating operator configuration: could not apply (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-multus/prometheus-k8s: failed to apply / update (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-multus/prometheus-k8s: Patch "https://api-int.ci-op-ydr5044s-c43d5.aws-2.ci.openshift.org:6443/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-multus/rolebindings/prometheus-k8s?fieldManager=cluster-network-operator%2Foperconfig&force=true": read tcp 10.0.148.39:43800->10.0.228.177:6443: read: connection reset by peer
May 10 01:14:32.304 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-232800: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9df58151b3ebc57a3f96a4013f9eb9aa expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3613a34b5f8ae0a40586cf967da2cde0, retrying]
May 10 01:14:32.304 - 491s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-232800: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9df58151b3ebc57a3f96a4013f9eb9aa expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3613a34b5f8ae0a40586cf967da2cde0, retrying]

... 1 lines not shown

May 09 22:53:00.497 E ns/e2e-k8s-sig-apps-job-upgrade-26 pod/foo-tqq7l node/ip-10-0-138-92.us-west-1.compute.internal uid/cf64a9a7-6bca-4620-a311-7912430f3328 container/c reason/ContainerExit code/137 cause/Error
May 09 22:54:29.693 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-151147: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f7b11efb8f012cd461da11aadb8e8fab expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-52646db6e7c0c986ecfd0127afb9d0ad, retrying]
May 09 22:54:29.693 - 502s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-151147: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f7b11efb8f012cd461da11aadb8e8fab expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-52646db6e7c0c986ecfd0127afb9d0ad, retrying]

... 1 lines not shown

May 09 17:13:24.892 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-scxff node/ip-10-0-179-152.us-west-2.compute.internal uid/1115d671-8837-425a-97d2-bffe4594acfb container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 09 17:14:38.051 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-151147: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c762b46ea9227657c099f9c7dfe0d864 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fccb828c304b98816a95b3dcb7177201, retrying]
May 09 17:14:38.051 - 138s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-151147: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c762b46ea9227657c099f9c7dfe0d864 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fccb828c304b98816a95b3dcb7177201, retrying]

... 1 lines not shown

May 09 15:33:52.768 E ns/openshift-monitoring pod/thanos-querier-6cbdb8cbd8-tfcx8 node/ip-10-0-222-204.us-west-2.compute.internal uid/a43a4036-72a5-4e0c-ae6f-883bcad774ab container/oauth-proxy reason/ContainerExit code/2 cause/Error 2024/05/09 15:04:34 provider.go:129: Defaulting client-id to system:serviceaccount:openshift-monitoring:thanos-querier\n2024/05/09 15:04:34 provider.go:134: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2024/05/09 15:04:34 provider.go:358: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2024/05/09 15:04:34 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:9090/"\n2024/05/09 15:04:34 oauthproxy.go:224: compiled skip-auth-regex => "^/-/(healthy|ready)$"\n2024/05/09 15:04:34 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:thanos-querier\n2024/05/09 15:04:34 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> samesite: refresh:disabled\n2024/05/09 15:04:34 http.go:107: HTTPS: listening on [::]:9091\nI0509 15:04:34.412861       1 dynamic_serving_content.go:130] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n
May 09 15:35:06.794 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-135203: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ffb48a8d1fea4bde6c05c85ecfbfcbcd expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-58589c40762484801201155c9903f615, retrying]
May 09 15:35:06.794 - 557s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-135203: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ffb48a8d1fea4bde6c05c85ecfbfcbcd expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-58589c40762484801201155c9903f615, retrying]

... 1 lines not shown

May 09 14:35:36.012 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-r9nj5 node/ip-10-0-233-223.ec2.internal uid/4cc39e2b-cdd6-4057-bcb6-45e45d84bb81 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 09 14:37:07.172 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-130339: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ab918506f10b4a2edd86289c23f5015a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a263dba52b92fa91a7e349da2ae583e1, retrying]
May 09 14:37:07.172 - 509s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-130339: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ab918506f10b4a2edd86289c23f5015a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a263dba52b92fa91a7e349da2ae583e1, retrying]

... 1 lines not shown

May 09 14:04:52.399 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-148-84.us-west-1.compute.internal uid/0471ead1-0c4e-406a-9df3-6de90b518e1d container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-09T14:04:49.235Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-09T14:04:49.235Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@2e06525f00f2, date=20240425-02:16:31)"\nts=2024-05-09T14:04:49.264Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-09T14:04:49.296Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-09T14:04:49.296Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-09T14:04:49.296Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=32.497346ms\n
May 09 14:05:29.305 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-120504: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f691c8878ae1357864b11347dc65e6d3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a958bded5961ba8079eebc28f1bdf986, retrying]
May 09 14:05:29.305 - 145s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-120504: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f691c8878ae1357864b11347dc65e6d3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a958bded5961ba8079eebc28f1bdf986, retrying]

... 1 lines not shown

May 09 13:10:55.450 E ns/e2e-k8s-sig-apps-job-upgrade-369 pod/foo-h8jfg node/ip-10-0-143-212.ec2.internal uid/ac66608d-572f-4e24-8995-3ff47c4fda51 container/c reason/ContainerExit code/137 cause/Error
May 09 13:12:26.121 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-111245: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-deeff0c52e3ce3a75da135d43aa00c08 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3951cd79dd196d92ddd8717a6cf40708, retrying]
May 09 13:12:26.121 - 516s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-111245: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-deeff0c52e3ce3a75da135d43aa00c08 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3951cd79dd196d92ddd8717a6cf40708, retrying]

... 1 lines not shown

May 09 11:45:44.807 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-143-50.ec2.internal uid/322412b1-6e1e-472d-bacd-a0f291dee1ee container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-09T11:45:41.573Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-09T11:45:41.573Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@2e06525f00f2, date=20240425-02:16:31)"\nts=2024-05-09T11:45:41.601Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-09T11:45:41.628Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-09T11:45:41.628Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-09T11:45:41.628Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=27.29823ms\n
May 09 11:47:47.833 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-100408: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fee0a68c7cc20187f8ef69f109edb67c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a72f3a073301c040cd1d342f9803569e, retrying]
May 09 11:47:47.833 - 106s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-100408: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fee0a68c7cc20187f8ef69f109edb67c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a72f3a073301c040cd1d342f9803569e, retrying]

... 1 lines not shown

May 09 10:45:54.578 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-161-17.ec2.internal uid/11aadff6-904c-444d-b9b2-e26cbc2b961c container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-09T10:45:51.944Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-09T10:45:51.944Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@2e06525f00f2, date=20240425-02:16:31)"\nts=2024-05-09T10:45:52.006Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-09T10:45:52.038Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-09T10:45:52.038Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-09T10:45:52.038Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=31.92666ms\n
May 09 10:47:54.779 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-085754: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a5d44e033fb86755dc9707081a476362 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6c4bb1b60b62922c500fe79c5a8cd802, retrying]
May 09 10:47:54.779 - 108s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-085754: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a5d44e033fb86755dc9707081a476362 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6c4bb1b60b62922c500fe79c5a8cd802, retrying]

... 1 lines not shown

May 09 09:32:18.988 - 35s   E clusteroperator/network condition/Degraded status/True reason/Error while updating operator configuration: could not apply (/v1, Kind=Service) openshift-multus/multus-admission-controller: failed to apply / update (/v1, Kind=Service) openshift-multus/multus-admission-controller: Patch "https://api-int.ci-op-rmdk2dif-c43d5.aws-2.ci.openshift.org:6443/api/v1/namespaces/openshift-multus/services/multus-admission-controller?fieldManager=cluster-network-operator%2Foperconfig&force=true": read tcp 10.0.151.186:40010->10.0.171.86:6443: read: connection reset by peer
May 09 09:33:00.152 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-075018: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9b0fd8002f770258d6c2ff6b6d673228 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-4fbd98d950a24afac81b92ebe9440c12, retrying]
May 09 09:33:00.152 - 497s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-075018: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9b0fd8002f770258d6c2ff6b6d673228 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-4fbd98d950a24afac81b92ebe9440c12, retrying]

... 1 lines not shown

May 09 08:43:53.773 - 7s    E clusteroperator/kube-storage-version-migrator condition/Available status/False reason/KubeStorageVersionMigratorAvailable: Waiting for Deployment
May 09 08:44:10.358 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-065322: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-579e2883edbe218c6a97a296de9fb80b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f241d6bcbe0702a38041bacad812b46c, retrying]
May 09 08:44:10.358 - 151s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-065322: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-579e2883edbe218c6a97a296de9fb80b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f241d6bcbe0702a38041bacad812b46c, retrying]

... 1 lines not shown

May 09 07:57:10.413 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-wxlh8 node/ip-10-0-168-235.us-west-2.compute.internal uid/8716ef46-3186-4cd0-a869-5fa0092285e0 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 09 07:58:34.310 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-054719: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-af8f46d4a82d83a8d28d113979e1ad7e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1d741f6bd497f71de976b859db84ce93, retrying]
May 09 07:58:34.310 - 537s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-054719: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-af8f46d4a82d83a8d28d113979e1ad7e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1d741f6bd497f71de976b859db84ce93, retrying]

... 1 lines not shown

May 09 04:30:14.694 E ns/e2e-k8s-sig-apps-job-upgrade-7661 pod/foo-xhv8s node/ip-10-0-143-137.us-east-2.compute.internal uid/5c165dd9-5d88-4281-a7f4-c9abf876da78 container/c reason/ContainerExit code/137 cause/Error
May 09 04:30:21.269 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-024720: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-00758111cf70efbda03028deb78f11df expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a8d1174197255662e2c727f9943cec5d, retrying]
May 09 04:30:21.269 - 132s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-024720: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-00758111cf70efbda03028deb78f11df expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a8d1174197255662e2c727f9943cec5d, retrying]

... 1 lines not shown

May 09 03:27:04.000 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-154-105.ec2.internal uid/9a517de1-9934-4c8b-b414-15c13735d8aa container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-09T03:27:01.407Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-09T03:27:01.407Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@2e06525f00f2, date=20240425-02:16:31)"\nts=2024-05-09T03:27:01.422Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-09T03:27:01.455Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-09T03:27:01.455Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-09T03:27:01.455Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=33.02405ms\n
May 09 03:28:57.158 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-014628: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1a05a9e12964b6f3c8a2d14ec3b9a6e3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1aac122802d8a95febeefa05023b5e62, retrying]
May 09 03:28:57.158 - 122s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-014628: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1a05a9e12964b6f3c8a2d14ec3b9a6e3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1aac122802d8a95febeefa05023b5e62, retrying]

... 1 lines not shown

May 09 02:23:19.562 E ns/e2e-k8s-sig-apps-job-upgrade-9080 pod/foo-fb5lr node/ip-10-0-152-234.us-east-2.compute.internal uid/4c668bc4-f41c-48c6-b3c8-1dc58baa02f1 container/c reason/ContainerExit code/137 cause/Error
May 09 02:24:20.964 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-09-004819: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f023fabcc29e4e567cdef97c2b8cf783 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-989dee6941cbc0abe7228ec6063899e9, retrying]
May 09 02:24:20.964 - 527s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-09-004819: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f023fabcc29e4e567cdef97c2b8cf783 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-989dee6941cbc0abe7228ec6063899e9, retrying]

... 1 lines not shown

May 09 01:36:26.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://af70086f3d65d4e73a0f9495c147623f-711785775.us-west-2.elb.amazonaws.com:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 09 01:36:33.411 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-234621: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-57d3b446f24cf4881b50f68a16fd6fe6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-41c5101525dd05bcf3226392a852d45c, retrying]
May 09 01:36:33.411 - 137s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-234621: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-57d3b446f24cf4881b50f68a16fd6fe6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-41c5101525dd05bcf3226392a852d45c, retrying]

... 1 lines not shown

May 09 00:54:53.640 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-151-111.ec2.internal uid/af305d2d-eb2d-4c49-a354-9eaa874b017e container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-09T00:54:49.998Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-09T00:54:49.998Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@2e06525f00f2, date=20240425-02:16:31)"\nts=2024-05-09T00:54:50.011Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-09T00:54:50.044Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-09T00:54:50.044Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-09T00:54:50.044Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=32.857899ms\n
May 09 00:56:38.967 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-225345: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-78d76654c2dd23ae341a83c398b656b7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-975dc70d6614b9ed5c224c48f10c01f2, retrying]
May 09 00:56:38.967 - 109s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-225345: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-78d76654c2dd23ae341a83c398b656b7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-975dc70d6614b9ed5c224c48f10c01f2, retrying]

... 1 lines not shown

May 08 23:29:00.984 E ns/e2e-k8s-sig-apps-job-upgrade-949 pod/foo-lpwhv node/ip-10-0-235-188.us-west-2.compute.internal uid/417b0735-6a08-4477-b1fb-074338e31eed container/c reason/ContainerExit code/137 cause/Error
May 08 23:30:02.387 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-214657: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-23594d5d16a2c55c8ab87359c1ba04c9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2fcd9c4a999dc194e644b0d40d487ab0, retrying]
May 08 23:30:02.387 - 517s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-214657: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-23594d5d16a2c55c8ab87359c1ba04c9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2fcd9c4a999dc194e644b0d40d487ab0, retrying]

... 1 lines not shown

May 08 22:31:48.000 - 5s    E ns/openshift-image-registry route/test-disruption-new disruption/image-registry connection/new reason/DisruptionBegan ns/openshift-image-registry route/test-disruption-new disruption/image-registry connection/new stopped responding to GET requests over new connections: Get "https://test-disruption-new-openshift-image-registry.apps.ci-op-rv3z6fi4-c43d5.aws-2.ci.openshift.org/healthz": EOF
May 08 22:32:28.611 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-204901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-632c6ec78df41dcf6443bc58c79f69bc expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3a07e07d0a35f849863d32cc70a53fac, retrying]
May 08 22:32:28.611 - 125s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-204901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-632c6ec78df41dcf6443bc58c79f69bc expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3a07e07d0a35f849863d32cc70a53fac, retrying]

... 1 lines not shown

May 08 21:48:10.141 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-168-144.us-west-1.compute.internal uid/7bbc75cc-bac4-496b-8f26-1d3e30d6974c container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T21:48:07.620Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-08T21:48:07.620Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@912fdea84186, date=20240425-02:16:52)"\nts=2024-05-08T21:48:07.637Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T21:48:07.673Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T21:48:07.673Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T21:48:07.673Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=35.933838ms\n
May 08 21:48:23.846 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-195007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8538bed005891bd424d67c29e0949f71 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fd20551a9ee535a2ba4a0d9661f0e99b, retrying]
May 08 21:48:23.846 - 156s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-195007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8538bed005891bd424d67c29e0949f71 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fd20551a9ee535a2ba4a0d9661f0e99b, retrying]

... 1 lines not shown

May 08 20:32:38.251 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-147-168.us-west-1.compute.internal uid/4ac87440-6f77-4d17-b55b-3e58e242a4f9 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T20:32:30.546Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-08T20:32:30.546Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@912fdea84186, date=20240425-02:16:52)"\nts=2024-05-08T20:32:30.634Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T20:32:30.666Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T20:32:30.666Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T20:32:30.666Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=32.226374ms\n
May 08 20:34:26.831 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-185237: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-473cb530b5ba5f2e027bf50397b8e73d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7dcb2526c1eda9a0e5a86384e59f081d, retrying]
May 08 20:34:26.831 - 530s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-185237: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-473cb530b5ba5f2e027bf50397b8e73d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7dcb2526c1eda9a0e5a86384e59f081d, retrying]

... 1 lines not shown

May 08 18:35:07.460 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-152-12.us-east-2.compute.internal uid/9be5ac94-350b-478a-b883-3eba16583144 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T18:35:04.521Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-08T18:35:04.521Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@2e06525f00f2, date=20240425-02:16:31)"\nts=2024-05-08T18:35:04.557Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T18:35:04.590Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T18:35:04.590Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T18:35:04.590Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=33.01428ms\n
May 08 18:36:39.883 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-165241: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-983ba9da0e94d745be3bec935222ff78 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0ee716a21b79c7f84d4a412715ff321f, retrying]
May 08 18:36:39.883 - 118s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-165241: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-983ba9da0e94d745be3bec935222ff78 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0ee716a21b79c7f84d4a412715ff321f, retrying]

... 1 lines not shown

May 08 19:30:32.501 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-191-200.us-east-2.compute.internal uid/64aa3b0d-79bf-4f37-9143-76d9d9467167 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-08T19:00:06.093364964Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=rhaos-4.12-rhel-8, revision=f8c084d)"\nlevel=info ts=2024-05-08T19:00:06.093430839Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20240508-00:15:05)"\nlevel=info ts=2024-05-08T19:00:06.09379317Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-08T19:00:10.236279037Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T19:00:10.236371233Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T19:00:14.036691698Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T19:01:23.697108139Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T19:02:47.383604003Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefil
May 08 19:32:08.699 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-175058: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4bfeee9b4620dc0aaf91bccdfca577c7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-60f4f35b1054c537dd681d60556322e6, retrying]
May 08 19:32:08.699 - 129s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-175058: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4bfeee9b4620dc0aaf91bccdfca577c7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-60f4f35b1054c537dd681d60556322e6, retrying]

... 1 lines not shown

May 08 15:26:35.157 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-164-80.us-west-2.compute.internal uid/749be137-8948-4255-bc3d-416a60f4ca6a container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T15:26:31.777Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-08T15:26:31.778Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@2e06525f00f2, date=20240425-02:16:31)"\nts=2024-05-08T15:26:31.866Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T15:26:31.897Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T15:26:31.898Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T15:26:31.898Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=31.166741ms\n
May 08 15:28:11.303 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-134139: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c31e85edf98f22ee9bb6f61dcc288c8d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1475c15ef8467ae23074f1516ee925e7, retrying]
May 08 15:28:11.303 - 139s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-134139: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c31e85edf98f22ee9bb6f61dcc288c8d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1475c15ef8467ae23074f1516ee925e7, retrying]

... 1 lines not shown

May 08 14:32:09.980 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-161-227.us-west-2.compute.internal uid/921c82d7-6499-4a03-9db3-9e6700372b76 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T14:32:03.694Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-08T14:32:03.694Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@912fdea84186, date=20240425-02:16:52)"\nts=2024-05-08T14:32:03.760Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T14:32:03.793Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T14:32:03.793Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T14:32:03.793Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=32.213496ms\n
May 08 14:32:49.532 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-124450: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8773425517677d4d30594be9522ac4e0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2c729030cf127d7ae84eaac92f5e8afd, retrying]
May 08 14:32:49.532 - 114s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-124450: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8773425517677d4d30594be9522ac4e0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2c729030cf127d7ae84eaac92f5e8afd, retrying]

... 1 lines not shown

May 08 06:44:05.972 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-190-28.us-west-2.compute.internal uid/61cdcdab-8a4f-43ea-82eb-703b473796fd container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T06:44:02.176Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-08T06:44:02.176Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@2e06525f00f2, date=20240425-02:16:31)"\nts=2024-05-08T06:44:02.189Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T06:44:02.221Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T06:44:02.221Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T06:44:02.221Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=31.455569ms\n
May 08 06:45:29.328 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-050139: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fb7b7418436ebe211ed4da5b939196b2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4f697fb43c70415495623baf9e6391a6, retrying]
May 08 06:45:29.328 - 145s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-050139: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fb7b7418436ebe211ed4da5b939196b2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4f697fb43c70415495623baf9e6391a6, retrying]

... 1 lines not shown

May 08 04:52:16.620 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-168-202.us-west-1.compute.internal uid/1a9623e0-c9b4-4c53-8724-dba71250c8c3 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T04:52:14.716Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=2742bd3a55bd39fee12a11daa9025fcae839c7dc)"\nts=2024-05-08T04:52:14.716Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@912fdea84186, date=20240425-02:16:52)"\nts=2024-05-08T04:52:14.744Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T04:52:14.768Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T04:52:14.768Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T04:52:14.768Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=23.598686ms\n
May 08 04:52:35.531 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-08-030302: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-33b4bec6a080acd6bb086bf9b949f363 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e8dec55539033e3eccbfeb2453380b67, retrying]
May 08 04:52:35.531 - 124s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-08-030302: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-33b4bec6a080acd6bb086bf9b949f363 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e8dec55539033e3eccbfeb2453380b67, retrying]

... 1 lines not shown

May 05 22:47:31.337 - 35s   E clusteroperator/network condition/Degraded status/True reason/Error while updating operator configuration: could not apply (apps/v1, Kind=Deployment) openshift-cloud-network-config-controller/cloud-network-config-controller: failed to apply / update (apps/v1, Kind=Deployment) openshift-cloud-network-config-controller/cloud-network-config-controller: Patch "https://api-int.ci-op-fbirdrlx-c43d5.aws-2.ci.openshift.org:6443/apis/apps/v1/namespaces/openshift-cloud-network-config-controller/deployments/cloud-network-config-controller?fieldManager=cluster-network-operator%2Foperconfig&force=true": read tcp 10.0.173.50:60158->10.0.158.145:6443: read: connection reset by peer
May 05 22:48:12.706 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-02-142700: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bee85c6011b7ba8584c9a14d1e980e4f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f84f51243481546f6416591f3b32d6df, retrying]
May 05 22:48:12.706 - 489s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-02-142700: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bee85c6011b7ba8584c9a14d1e980e4f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f84f51243481546f6416591f3b32d6df, retrying]

... 1 lines not shown