May 15 08:39:25.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://35.224.184.164:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 15 08:40:20.031 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-15-065158: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a8850cffbe791fb1b5dd15981587b74c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-baafb60824f2470e7c23f2ae9a917e28, retrying]
May 15 08:40:20.031 - 408s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-15-065158: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a8850cffbe791fb1b5dd15981587b74c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-baafb60824f2470e7c23f2ae9a917e28, retrying]

... 1 lines not shown

May 15 22:23:32.222 E ns/openshift-multus pod/multus-additional-cni-plugins-dvrfq node/ci-op-tr3ls4x2-7a89c-2ndv9-worker-c-l256w uid/e3bd7123-672b-4207-ae67-6906db646b6e container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 22:24:26.684 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-15-204102: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6dc106d14c29084b17d09caffcc7ad84 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-d0d182ae7a6bb7ef481eac3508a7ad17, retrying]
May 15 22:24:26.684 - 96s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-15-204102: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6dc106d14c29084b17d09caffcc7ad84 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-d0d182ae7a6bb7ef481eac3508a7ad17, retrying]

... 1 lines not shown

May 14 15:11:43.983 E ns/openshift-e2e-loki pod/loki-promtail-5hscs node/ci-op-4tvb0glg-7a89c-qcg7z-worker-c-g4hj2 uid/27c2fb69-0177-4526-bc84-47bca5035093 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 15:12:33.519 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-14-132144: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e312480587b81f10bab983bb546089b8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4f6055f0646ec6eb6e624a468f53e236, retrying]
May 14 15:12:33.519 - 93s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-14-132144: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e312480587b81f10bab983bb546089b8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4f6055f0646ec6eb6e624a468f53e236, retrying]

... 1 lines not shown

May 11 16:22:53.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://34.28.244.44:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 11 16:23:44.695 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-11-142650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6fcbd160b8778f9262a7171565589c86 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-0f9095a57b1c9cd10be396458af64b8e, retrying]
May 11 16:23:44.695 - 456s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-11-142650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6fcbd160b8778f9262a7171565589c86 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-0f9095a57b1c9cd10be396458af64b8e, retrying]

... 1 lines not shown

May 11 10:13:34.816 E ns/openshift-multus pod/network-metrics-daemon-6fxd8 node/ci-op-lnrhzpbx-7a89c-kdq8g-worker-c-4qlm6 uid/686332b0-9492-43f5-863f-2ded93f2c1c9 container/kube-rbac-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 11 10:15:28.062 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-11-082650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d498fb8e36295b1c98136d3a4e36ef67 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bf6483942324c8998d16d5f917672155, retrying]
May 11 10:15:28.062 - 34s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-11-082650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d498fb8e36295b1c98136d3a4e36ef67 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bf6483942324c8998d16d5f917672155, retrying]

... 1 lines not shown

May 14 23:20:14.930 E ns/openshift-e2e-loki pod/loki-promtail-pzpn7 node/ci-op-i9b7mksz-7a89c-2kqsz-worker-b-4d9x5 uid/4dc6eabc-7e72-40e3-80b7-ef77f69af13c container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 23:20:35.036 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-14-213601: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-da3d086245922f997537d0fde3a9a660 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fab333fd2468deed375011c4df990bcc, retrying]
May 14 23:20:35.036 - 445s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-14-213601: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-da3d086245922f997537d0fde3a9a660 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fab333fd2468deed375011c4df990bcc, retrying]

... 1 lines not shown

May 11 09:45:13.158 E ns/openshift-e2e-loki pod/loki-promtail-htrnc node/ci-op-qp60hbip-7a89c-dv2nr-worker-c-s4j5z uid/a717cedf-b7d7-43e3-b065-c35ce65e96ad container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 11 09:46:10.910 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-45a0a990e6a481011e550a940bd4b5c3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-25d2c28f5b3d7b75f528d70011004a59, retrying]
May 11 09:46:10.910 - 78s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-45a0a990e6a481011e550a940bd4b5c3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-25d2c28f5b3d7b75f528d70011004a59, retrying]

... 1 lines not shown

May 16 16:38:19.461 E ns/openshift-monitoring pod/alertmanager-main-1 node/ci-op-6wt7d4g9-7a89c-w866l-worker-b-4wmk2 uid/3aeb2426-02c7-4421-8b4f-c2acb542fa3c container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-16T16:38:15.039Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.12-rhel-8, revision=85c9b46fcf2890772ad3c7f7bdbeaadb7a1f5273)"\nts=2024-05-16T16:38:15.039Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@4cb74412faf8, date=20240514-16:39:22)"\nts=2024-05-16T16:38:15.247Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-16T16:38:15.338Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-16T16:38:15.338Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-16T16:38:15.338Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=90.530565ms\n
May 16 16:38:32.002 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-16-144840: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-00c94a01e689a9482b7bb88a0c2f77d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3722aeb7223bf3f327363e960dd440ff, retrying]
May 16 16:38:32.002 - 464s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-16-144840: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-00c94a01e689a9482b7bb88a0c2f77d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3722aeb7223bf3f327363e960dd440ff, retrying]

... 1 lines not shown

May 13 09:50:31.417 E ns/openshift-e2e-loki pod/loki-promtail-dr7gm node/ci-op-htj8tg2r-7a89c-w8t2f-worker-c-kxgk8 uid/a238cc35-ae6c-4a0f-b204-1fcfd4a86669 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 13 09:52:36.194 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-aa9f3178fa50444a810538c73945b8ad expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-153510c6c9953897b414e8c05bbad92b, retrying]
May 13 09:52:36.194 - 92s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-aa9f3178fa50444a810538c73945b8ad expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-153510c6c9953897b414e8c05bbad92b, retrying]

... 1 lines not shown

May 10 22:38:33.061 - 1s    E node/ci-op-0rz2tk13-7a89c-qgjrl-master-1 reason/FailedToDeleteCGroupsPath May 10 22:38:33.061779 ci-op-0rz2tk13-7a89c-qgjrl-master-1 hyperkube[2117]: I0510 22:38:33.061650    2117 pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods burstable podb02d93429b991edfa11c919721308dd7] err="unable to destroy cgroup paths for cgroup [kubepods burstable podb02d93429b991edfa11c919721308dd7] : Timed out while waiting for systemd to remove kubepods-burstable-podb02d93429b991edfa11c919721308dd7.slice"
May 10 22:40:12.320 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.57: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0e225f8dceddec8ea4e7daff3e60ff40 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-de2d5774bd79a6d9e8b8872f5b2429e7, retrying]
May 10 22:40:12.320 - 483s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.57: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0e225f8dceddec8ea4e7daff3e60ff40 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-de2d5774bd79a6d9e8b8872f5b2429e7, retrying]

... 1 lines not shown

May 11 04:06:36.430 E ns/e2e-k8s-sig-apps-job-upgrade-6032 pod/foo-rghlj node/ci-op-z1xlvmpl-7a89c-tn59m-worker-c-gzjcr uid/63012568-d323-41c1-af84-3296435401de container/c reason/ContainerExit code/137 cause/Error
May 11 04:06:53.804 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-11-022650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b203220ab3f3804cc0a971393f4b8e0f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3427c583375a6cf13216ab69ac1c3f05, retrying]
May 11 04:06:53.804 - 435s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-11-022650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b203220ab3f3804cc0a971393f4b8e0f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3427c583375a6cf13216ab69ac1c3f05, retrying]

... 1 lines not shown

May 10 17:31:43.259 E ns/openshift-e2e-loki pod/loki-promtail-szm44 node/ci-op-pnjd5b25-7a89c-7dgcv-worker-c-7p9sd uid/9804f71e-1c3c-41c6-a944-d46380065bc1 container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 17:33:03.013 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-10-124007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bff7c488eafb7bc3bf07cbdc3b1bccde expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3598eb3d2139befee661f546888e317d, retrying]
May 10 17:33:03.013 - 84s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-10-124007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bff7c488eafb7bc3bf07cbdc3b1bccde expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3598eb3d2139befee661f546888e317d, retrying]

... 1 lines not shown

May 10 14:27:32.188 E ns/openshift-e2e-loki pod/loki-promtail-kdtn8 node/ci-op-c8nxrtiv-7a89c-dxjfv-worker-c-4xwct uid/7be6e744-dfc7-4586-84a2-27836bec84e6 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 14:29:08.626 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-10-124007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-13908b7ffe333f3ebdd83c453f5dbbcf expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5b19b705eb56904219240d886f28e608, retrying]
May 10 14:29:08.626 - 150s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-10-124007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-13908b7ffe333f3ebdd83c453f5dbbcf expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5b19b705eb56904219240d886f28e608, retrying]

... 1 lines not shown

May 10 07:12:59.463 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ci-op-q1kjjn9d-7a89c-zg4c6-worker-c-zndr7 uid/55150667-4644-43b6-a709-be0349689ce5 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-10T06:37:39.613694052Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=rhaos-4.12-rhel-8, revision=a2a6d69)"\nlevel=info ts=2024-05-10T06:37:39.613799239Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20240509-20:14:50)"\nlevel=info ts=2024-05-10T06:37:39.614200205Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-10T06:37:48.294747176Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T06:37:48.294885435Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T06:37:49.537348324Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T06:40:23.634975474Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T06:45:34.692930036Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefi
May 10 07:13:00.932 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-10-052846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2483ab9c07c4a4791b1ad63ac18e627 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-853c5821f9cf51db3f52fd09fc2bbd85, retrying]
May 10 07:13:00.932 - 425s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-10-052846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2483ab9c07c4a4791b1ad63ac18e627 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-853c5821f9cf51db3f52fd09fc2bbd85, retrying]

... 1 lines not shown

May 10 01:10:39.822 E ns/openshift-e2e-loki pod/loki-promtail-cws87 node/ci-op-vkdzcji3-7a89c-d4tz5-worker-b-cvzmr uid/32148bc2-7ee0-42b7-8d24-fd55de6bba38 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 01:10:58.839 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-09-232846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9e46dda3eb8ab38f253d7789ad755611 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-cc3af5cefcfe28ddb0ab933fa2b0afc0, retrying]
May 10 01:10:58.839 - 446s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-09-232846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9e46dda3eb8ab38f253d7789ad755611 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-cc3af5cefcfe28ddb0ab933fa2b0afc0, retrying]

... 1 lines not shown

May 09 17:02:55.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://104.154.202.231:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 09 17:03:17.022 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-722105ac3110073155d12b40b00e585d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f06883347b85fd48c4b50a29baded9ca, retrying]
May 09 17:03:17.022 - 75s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-722105ac3110073155d12b40b00e585d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f06883347b85fd48c4b50a29baded9ca, retrying]

... 1 lines not shown

May 09 14:11:16.851 E ns/openshift-e2e-loki pod/loki-promtail-jrfs7 node/ci-op-jjnpz3mx-7a89c-qswkg-worker-c-25n6r uid/afd86ca0-5faf-4bb5-9b29-8658c49f1266 container/promtail reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 14:11:35.748 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d877411240ac287e67b04dca665dcd86 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-34f294b1a91e0c6e5344e07f0daf7b9b, retrying]
May 09 14:11:35.748 - 87s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d877411240ac287e67b04dca665dcd86 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-34f294b1a91e0c6e5344e07f0daf7b9b, retrying]

... 1 lines not shown

May 09 09:58:24.049 E ns/openshift-e2e-loki pod/loki-promtail-zpf4q node/ci-op-mrl6gmpi-7a89c-8qnwv-worker-c-fx99k uid/97b29a12-47cb-453a-95af-cc3bcc261f17 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 09:59:11.631 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9b4d7c899218a37dfa7223424b7b59fe expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bd566f1a787a3a75cfdbf1dc7b6207c7, retrying]
May 09 09:59:11.631 - 79s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9b4d7c899218a37dfa7223424b7b59fe expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bd566f1a787a3a75cfdbf1dc7b6207c7, retrying]

... 1 lines not shown

May 08 04:44:52.456 E ns/openshift-e2e-loki pod/loki-promtail-hlfts node/ci-op-vzd3fqdi-7a89c-g2zbj-worker-b-6q59x uid/708efd16-56cd-417c-a1b4-a1f96e35bdc3 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 04:45:11.121 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-08-030344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4b4ee47a4fff86c9f69f60bfc3c76e06 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-04c8fec91069be3f6ae77b7b0e814ec6, retrying]
May 08 04:45:11.121 - 463s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-08-030344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4b4ee47a4fff86c9f69f60bfc3c76e06 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-04c8fec91069be3f6ae77b7b0e814ec6, retrying]

... 1 lines not shown

May 08 14:36:58.895 E clusterversion/version changed Failing to True: ClusterOperatorNotAvailable: Cluster operator operator-lifecycle-manager-packageserver is not available
May 08 14:37:01.753 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-08-124615: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f628ad8fe2a59566b813b8393e194cc3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-def4938b890072b58a8d69a94075b732, retrying]
May 08 14:37:01.753 - 99s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-08-124615: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f628ad8fe2a59566b813b8393e194cc3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-def4938b890072b58a8d69a94075b732, retrying]

... 1 lines not shown

May 07 09:38:51.848 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-6ltbwwc8-7a89c-hhhc9-worker-c-k9wvv uid/0245211e-dd2c-423a-bbfe-51229ab063bd container/alertmanager reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 07 09:38:59.695 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-64c34f85650c939ecd95e4363122f053 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6cbf29c7e5179fd1309cedecd7ebb975, retrying]
May 07 09:38:59.695 - 424s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-64c34f85650c939ecd95e4363122f053 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6cbf29c7e5179fd1309cedecd7ebb975, retrying]

... 1 lines not shown

May 05 09:41:25.631 E ns/openshift-e2e-loki pod/loki-promtail-nmwqp node/ci-op-bi1qsfll-7a89c-w2tdv-worker-b-xzg52 uid/b7c79fdc-0f0b-4176-8141-ab3f30093ab3 container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 05 09:41:39.497 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d0f50dcf8508b9841281f3d97690a8ed expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-34ddca42a4cddb9d44805ef15118943e, retrying]
May 05 09:41:39.497 - 449s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d0f50dcf8508b9841281f3d97690a8ed expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-34ddca42a4cddb9d44805ef15118943e, retrying]

... 1 lines not shown

May 03 09:45:12.835 E ns/openshift-e2e-loki pod/loki-promtail-j24wm node/ci-op-3l9lmnv2-7a89c-45q4h-worker-b-fnzx7 uid/d91f92a5-7d48-4025-86e2-840ee0beb117 container/promtail reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 03 09:45:23.588 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-afcf04f9a6bed6db1168b971f94f9eee expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-10d43aa6de296edaaacf81fa66803f9c, retrying]
May 03 09:45:23.588 - 443s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-afcf04f9a6bed6db1168b971f94f9eee expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-10d43aa6de296edaaacf81fa66803f9c, retrying]

... 1 lines not shown