Job:
periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-aws-sdn-upgrade (all) - 53 runs, 51% failed, 7% of failures match = 4% impact
#1620255981558566912junit5 hours ago
Jan 31 03:40:54.187 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 31 03:40:54.187 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 31 03:52:28.665 E ns/openshift-kube-scheduler-operator pod/openshift-kube-scheduler-operator-7db565db9-vjx9z node/ip-10-0-196-236.us-west-1.compute.internal uid/a1c5973d-c0a6-40d6-bd4c-02bb2a469619 container/kube-scheduler-operator-container reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
#1620255981558566912junit5 hours ago
Jan 31 03:40:54.187 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
#1616133293483233280junit11 days ago
Jan 19 19:39:58.449 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 19 19:39:58.449 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 19 19:42:39.422 E ns/openshift-image-registry pod/node-ca-rftt6 node/ip-10-0-207-115.us-west-1.compute.internal uid/ac4808f8-8265-470e-aca6-df60fd55269c container/node-ca reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
#1616133293483233280junit11 days ago
Jan 19 19:39:58.449 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-aws-upgrade (all) - 24 runs, 50% failed, 8% of failures match = 4% impact
#1620254489896292352junit5 hours ago
Jan 31 04:36:29.253 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 31 04:36:29.253 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 31 04:36:29.936 E ns/openshift-ingress-canary pod/ingress-canary-llr9s node/ip-10-0-131-51.us-east-2.compute.internal container/serve-healthcheck-canary reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
#1620254489896292352junit5 hours ago
Jan 31 04:36:29.253 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-stable-4.12-ocp-e2e-aws-heterogeneous-upgrade (all) - 74 runs, 38% failed, 4% of failures match = 1% impact
#1620237127398723584junit7 hours ago
Jan 31 03:38:49.134 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: Get "https://172.30.0.1:443/apis/apiregistration.k8s.io/v1/apiservices/v1.apps.openshift.io": read tcp 10.128.0.72:52536->172.30.0.1:443: read: connection timed out
Jan 31 03:38:49.134 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: Get "https://172.30.0.1:443/apis/apiregistration.k8s.io/v1/apiservices/v1.apps.openshift.io": read tcp 10.128.0.72:52536->172.30.0.1:443: read: connection timed out
Jan 31 03:38:52.000 - 1s    E disruption/service-load-balancer-with-pdb connection/new reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/new stopped responding to GET requests over new connections: Get "http://a4650f7dec9814864ab8bdb4653714e9-201905393.us-east-2.elb.amazonaws.com:80/echo?msg=Hello": net/http: timeout awaiting response headers
#1620237127398723584junit7 hours ago
Jan 31 03:38:49.134 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: Get "https://172.30.0.1:443/apis/apiregistration.k8s.io/v1/apiservices/v1.apps.openshift.io": read tcp 10.128.0.72:52536->172.30.0.1:443: read: connection timed out
periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.12-upgrade-from-stable-4.11-e2e-openstack-sdn-upgrade (all) - 7 runs, 100% failed, 43% of failures match = 43% impact
#1620209212502052864junit8 hours ago
Jan 31 02:00:39.471 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 31 02:00:39.471 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 31 02:01:19.348 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2023-01-27-052652: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6c1f2b372f2ac048c8a18fff82a2f0c7 expected e3dc9430ac753ac0440f55910370da22c65637f1 has b36482885ba1304e122e7c01c26cd671dfdd0418: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-dc3f1c4f30f63e9f355f397f7ab10617, retrying]
#1620209212502052864junit8 hours ago
Jan 31 02:00:39.471 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
1 tests failed during this blip (2023-01-31 02:00:39.471413694 +0000 UTC to 2023-01-31 02:00:39.471413694 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1616311806790209536junit11 days ago
Jan 20 07:37:03.471 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 20 07:37:03.471 - 27ms  E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 20 07:37:03.498 E clusteroperator/authentication condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.oauth.openshift.io: not available: failing or missing response from https://10.130.0.57:8443/apis/oauth.openshift.io/v1: Get "https://10.130.0.57:8443/apis/oauth.openshift.io/v1": dial tcp 10.130.0.57:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.user.openshift.io: not available: failing or missing response from https://10.130.0.57:8443/apis/user.openshift.io/v1: Get "https://10.130.0.57:8443/apis/user.openshift.io/v1": dial tcp 10.130.0.57:8443: connect: connection refused
#1616311806790209536junit11 days ago
Jan 20 07:37:03.471 - 27ms  E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
1 tests failed during this blip (2023-01-20 07:37:03.471709082 +0000 UTC to 2023-01-20 07:37:03.471709082 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1615526874895618048junit13 days ago
Jan 18 04:06:59.751 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 18 04:06:59.751 - 986ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 18 04:07:10.487 - 1s    E disruption/kube-api connection/new reason/DisruptionBegan disruption/kube-api connection/new stopped responding to GET requests over new connections: Get "https://api.qbkcmxql-ae6d2.shiftstack.devcluster.openshift.com:6443/api/v1/namespaces/default": EOF
#1615526874895618048junit13 days ago
Jan 18 04:10:35.073 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.105:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.105:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.105:8443: i/o timeout
Jan 18 04:10:35.073 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.105:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.105:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.105:8443: i/o timeout
Jan 18 04:10:35.487 - 2s    E disruption/openshift-api connection/new reason/DisruptionBegan disruption/openshift-api connection/new stopped responding to GET requests over new connections: Get "https://api.qbkcmxql-ae6d2.shiftstack.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/default/imagestreams": net/http: timeout awaiting response headers
#1615526874895618048junit13 days ago
Jan 18 04:16:37.100 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.129.0.105:8443/apis/authorization.openshift.io/v1: Get "https://10.129.0.105:8443/apis/authorization.openshift.io/v1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 18 04:16:37.100 - 8s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.129.0.105:8443/apis/authorization.openshift.io/v1: Get "https://10.129.0.105:8443/apis/authorization.openshift.io/v1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 18 04:16:37.487 - 999ms E disruption/cache-oauth-api connection/new reason/DisruptionBegan disruption/cache-oauth-api connection/new stopped responding to GET requests over new connections: Get "https://api.qbkcmxql-ae6d2.shiftstack.devcluster.openshift.com:6443/apis/oauth.openshift.io/v1/oauthclients?resourceVersion=0": net/http: timeout awaiting response headers
#1615526874895618048junit13 days ago
Jan 18 04:19:18.483 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 18 04:19:18.483 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 18 04:19:20.129 E ns/openshift-e2e-loki pod/loki-promtail-j66wn node/qbkcmxql-ae6d2-ch74x-worker-0-sshb9 uid/f3bf2688-92a6-43a7-aeda-e8759037ec0e container/oauth-proxy reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
#1615526874895618048junit13 days ago
1 tests failed during this blip (2023-01-18 03:11:09.375069597 +0000 UTC to 2023-01-18 03:11:09.375069597 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
Jan 18 04:06:59.751 - 986ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
1 tests failed during this blip (2023-01-18 04:06:59.751512579 +0000 UTC to 2023-01-18 04:06:59.751512579 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
Jan 18 04:10:35.073 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.105:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.105:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.105:8443: i/o timeout
1 tests failed during this blip (2023-01-18 04:10:35.073326801 +0000 UTC to 2023-01-18 04:10:35.073326801 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-azure-upgrade (all) - 67 runs, 85% failed, 9% of failures match = 7% impact
#1620049783202779136junit18 hours ago
Jan 30 14:39:15.118 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field
Jan 30 14:39:15.118 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field
Jan 30 14:41:18.006 E clusterversion/version changed Failing to True: MultipleErrors: Multiple errors are preventing progress:\n* Cluster operator etcd is updating versions\n* Cluster operator kube-apiserver is updating versions
#1620049783202779136junit18 hours ago
Jan 30 14:39:15.118 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field
#1618673008354791424junit4 days ago
Jan 26 19:47:50.650 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 19:47:50.650 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 19:47:51.721 E ns/openshift-sdn pod/sdn-dn7xl node/ci-op-i5dqx507-253f3-wm9bw-worker-centralus1-kvhq2 container/drop-icmp reason/ContainerExit code/143 cause/Error
#1618673008354791424junit4 days ago
Jan 26 19:47:50.650 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
1 tests failed during this blip (2023-01-26 19:47:50.650675991 +0000 UTC to 2023-01-26 19:47:50.650675991 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1618342689005113344junit5 days ago
Jan 25 21:49:55.572 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 25 21:49:55.572 - 14s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 25 21:50:01.663 E ns/openshift-sdn pod/sdn-vc2l9 node/ci-op-cpm8hkyf-253f3-zslkv-master-0 container/sdn reason/ContainerExit code/137 cause/Error penshift-ovs-networkpolicy"\nI0125 21:49:29.728505  107501 cmd.go:159] Starting node networking (4.9.0-202212051626.p0.g006ee16.assembly.stream-006ee16)\nI0125 21:49:29.728539  107501 node.go:299] Starting openshift-sdn network plugin\nI0125 21:49:29.998851  107501 healthcheck_ovs.go:18] Starting OVS health check\nI0125 21:49:29.999580  107501 healthcheck_ovs.go:30] Starting SDN-OVS disconnection go-routine\nI0125 21:49:30.044961  107501 sdn_controller.go:135] [SDN setup] SDN is already set up\nI0125 21:49:30.165197  107501 node.go:348] Starting openshift-sdn pod manager\nI0125 21:49:30.194336  107501 node.go:242] Checking default interface MTU\nI0125 21:49:30.209274  107501 networkpolicy.go:362] SyncVNIDRules: 0 unused VNIDs\nI0125 21:49:30.210400  107501 proxy.go:117] Using unidling+iptables Proxier.\nI0125 21:49:30.210688  107501 proxier.go:293] "Using iptables mark for masquerade" ipFamily=IPv4 mark="0x00000001"\nI0125 21:49:30.210751  107501 proxier.go:339] "Iptables sync params" ipFamily=IPv4 minSyncPeriod="1s" syncPeriod="30s" burstSyncs=2\nI0125 21:49:30.210823  107501 proxier.go:349] "Iptables supports --random-fully" ipFamily=IPv4\nI0125 21:49:30.211028  107501 proxy.go:157] Tearing down userspace rules.\nI0125 21:49:30.378781  107501 proxier.go:247] "Setting proxy IP and initializing iptables" ip="10.0.0.7"\ninterrupt: Signal terminated received. Gracefully shutting down ...\nI0125 21:49:30.548774  107501 proxy.go:110] Starting multitenant SDN proxy endpoint filter\nI0125 21:49:30.558727  107501 config.go:315] Starting service config controller\nI0125 21:49:30.558743  107501 config.go:224] Starting endpoint slice config controller\nI0125 21:49:30.558768  107501 shared_informer.go:240] Waiting for caches to sync for endpoint slice config\nI0125 21:49:30.558768  107501 shared_informer.go:240] Waiting for caches to sync for service config\nI0125 21:49:30.564887  107501 proxy.go:248] Started Kubernetes Proxy on 0.0.0.0\nI0125 21:49:30.565066  107501 cmd.go:170] openshift-sdn network plugin waiting for proxy startup to complete\n
#1618342689005113344junit5 days ago
Jan 25 21:49:55.572 - 14s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
1 tests failed during this blip (2023-01-25 21:49:55.572770658 +0000 UTC to 2023-01-25 21:49:55.572770658 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1617713603098447872junit7 days ago
Jan 24 04:28:52.249 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 24 04:28:52.249 - 5s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 24 04:28:52.470 E clusteroperator/authentication condition/Available status/False reason/OAuthServerRouteEndpointAccessibleController_EndpointUnavailable changed: OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.ci-op-0x9jtmy3-253f3.ci.azure.devcluster.openshift.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
#1617713603098447872junit7 days ago
Jan 24 04:28:52.249 - 5s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
#1615619882538242048junit13 days ago
Jan 18 09:33:23.787 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 18 09:33:23.787 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 18 09:33:24.000 - 1s    E ns/openshift-console route/console disruption/ingress-to-console connection/new ns/openshift-console route/console disruption/ingress-to-console connection/new stopped responding to GET requests over new connections: Get "https://console-openshift-console.apps.ci-op-ztwqin6n-253f3.ci.azure.devcluster.openshift.com/healthz": net/http: timeout awaiting response headers
#1615619882538242048junit13 days ago
Jan 18 09:33:23.787 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-aws-sdn-upgrade-infra (all) - 5 runs, 100% failed, 20% of failures match = 20% impact
#1619743818100248576junit40 hours ago
Jan 29 18:37:28.663 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 29 18:37:28.663 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 29 18:37:31.835 E ns/openshift-sdn pod/sdn-kgrrs node/ip-10-0-150-233.us-west-1.compute.internal uid/dfa23a25-1d84-452d-a927-58e02f3381dc container/sdn reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
#1619743818100248576junit40 hours ago
Jan 29 18:37:28.663 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-azure-sdn-upgrade (all) - 342 runs, 73% failed, 3% of failures match = 2% impact
#1619222059567550464junit3 days ago
Jan 28 08:32:24.269 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 28 08:32:24.269 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 28 08:32:35.305 - 1s    E disruption/cache-oauth-api connection/new reason/DisruptionBegan disruption/cache-oauth-api connection/new stopped responding to GET requests over new connections: Get "https://api.ci-op-4w83hp91-7f65d.ci2.azure.devcluster.openshift.com:6443/apis/oauth.openshift.io/v1/oauthclients?resourceVersion=0": dial tcp 20.237.180.233:6443: i/o timeout
#1619222059567550464junit3 days ago
Jan 28 08:32:24.269 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
1 tests failed during this blip (2023-01-28 08:32:24.269604274 +0000 UTC to 2023-01-28 08:32:24.269604274 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1618360654052200448junit5 days ago
Jan 25 23:10:24.815 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.130.0.94:8443/apis/apps.openshift.io/v1: Get "https://10.130.0.94:8443/apis/apps.openshift.io/v1": dial tcp 10.130.0.94:8443: connect: no route to host
Jan 25 23:10:24.815 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.130.0.94:8443/apis/apps.openshift.io/v1: Get "https://10.130.0.94:8443/apis/apps.openshift.io/v1": dial tcp 10.130.0.94:8443: connect: no route to host
Jan 25 23:10:30.418 E ns/openshift-cluster-csi-drivers pod/azure-file-csi-driver-operator-64d647844b-tp6zt node/ci-op-bi6jbtjr-7f65d-9tq6m-master-0 uid/4ba7368b-8b9d-4bc8-a94a-b48cadab8d30 container/azure-file-csi-driver-operator reason/ContainerExit code/1 cause/Error
#1618360654052200448junit5 days ago
Jan 25 23:10:24.815 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.130.0.94:8443/apis/apps.openshift.io/v1: Get "https://10.130.0.94:8443/apis/apps.openshift.io/v1": dial tcp 10.130.0.94:8443: connect: no route to host
#1616770802441523200junit9 days ago
Jan 21 13:37:57.389 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 13:37:57.389 - 92ms  E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 13:38:29.286 E ns/openshift-etcd pod/etcd-ci-op-8wdqljh2-7f65d-d8cwp-master-0 node/ci-op-8wdqljh2-7f65d-d8cwp-master-0 uid/e748d7ca-9a2e-41d8-a719-dae95a5908fa invariant violation: container statuses were removed
#1616770802441523200junit9 days ago
Jan 21 13:37:57.389 - 92ms  E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
1 tests failed during this blip (2023-01-21 13:37:57.389809524 +0000 UTC to 2023-01-21 13:37:57.389809524 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1616770802571546624junit9 days ago
Jan 21 13:14:47.005 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 13:14:47.005 - 2s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 13:16:30.651 E ns/openshift-kube-controller-manager pod/installer-9-ci-op-xxk5ky4l-7f65d-zwgl5-master-0 node/ci-op-xxk5ky4l-7f65d-zwgl5-master-0 uid/998fe785-e8f2-4b31-a586-cd4773c89af2 container/installer reason/ContainerExit code/1 cause/Error MapNamePrefixes: ([]string) (len=8 cap=8) {\n  (string) (len=27) "kube-controller-manager-pod",\n  (string) (len=6) "config",\n  (string) (len=32) "cluster-policy-controller-config",\n  (string) (len=29) "controller-manager-kubeconfig",\n  (string) (len=38) "kube-controller-cert-syncer-kubeconfig",\n  (string) (len=17) "serviceaccount-ca",\n  (string) (len=10) "service-ca",\n  (string) (len=15) "recycler-config"\n },\n OptionalConfigMapNamePrefixes: ([]string) (len=1 cap=1) {\n  (string) (len=12) "cloud-config"\n },\n CertSecretNames: ([]string) (len=2 cap=2) {\n  (string) (len=39) "kube-controller-manager-client-cert-key",\n  (string) (len=10) "csr-signer"\n },\n OptionalCertSecretNamePrefixes: ([]string) <nil>,\n CertConfigMapNamePrefixes: ([]string) (len=2 cap=2) {\n  (string) (len=20) "aggregator-client-ca",\n  (string) (len=9) "client-ca"\n },\n OptionalCertConfigMapNamePrefixes: ([]string) (len=1 cap=1) {\n  (string) (len=17) "trusted-ca-bundle"\n },\n CertDir: (string) (len=66) "/etc/kubernetes/static-pod-resources/kube-controller-manager-certs",\n ResourceDir: (string) (len=36) "/etc/kubernetes/static-pod-resources",\n PodManifestDir: (string) (len=25) "/etc/kubernetes/manifests",\n Timeout: (time.Duration) 2m0s,\n StaticPodManifestsLockFile: (string) "",\n PodMutationFns: ([]installerpod.PodMutationFunc) <nil>,\n KubeletVersion: (string) ""\n})\nI0121 13:15:59.280385       1 cmd.go:410] Getting controller reference for node ci-op-xxk5ky4l-7f65d-zwgl5-master-0\nI0121 13:15:59.295131       1 cmd.go:423] Waiting for installer revisions to settle for node ci-op-xxk5ky4l-7f65d-zwgl5-master-0\nI0121 13:15:59.310984       1 cmd.go:515] Waiting additional period after revisions have settled for node ci-op-xxk5ky4l-7f65d-zwgl5-master-0\nI0121 13:16:06.280898       1 cmd.go:124] Received SIGTERM or SIGINT signal, shutting down the process.\nI0121 13:16:29.311169       1 cmd.go:521] Getting installer pods for node ci-op-xxk5ky4l-7f65d-zwgl5-master-0\nF0121 13:16:29.311314       1 cmd.go:106] client rate limiter Wait returned an error: context canceled\n
#1616770802571546624junit9 days ago
Jan 21 13:14:47.005 - 2s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
1 tests failed during this blip (2023-01-21 13:14:47.005475719 +0000 UTC to 2023-01-21 13:14:47.005475719 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1616680217718296576junit10 days ago
Jan 21 09:25:45.520 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:25:45.520 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:25:46.170 - 998ms E disruption/openshift-api connection/reused reason/DisruptionBegan disruption/openshift-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.8:8443: connect: connection refused\n
#1616680217718296576junit10 days ago
Jan 21 09:26:03.531 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:26:03.531 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:26:04.167 - 999ms E disruption/openshift-api connection/new reason/DisruptionBegan disruption/openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.8:8443: connect: connection refused\n
#1616680217718296576junit10 days ago
Jan 21 09:26:48.533 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:26:48.533 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:26:49.167 - 2s    E disruption/openshift-api connection/new reason/DisruptionBegan disruption/openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.8:8443: connect: connection refused\n
#1616680217718296576junit10 days ago
Jan 21 09:27:24.518 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:27:24.518 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:27:30.186 E ns/e2e-ephemeral-4694-792 pod/csi-hostpathplugin-0 node/ci-op-it9m1t2c-7f65d-dbt7m-worker-centralus2-mnjlf uid/51241bd7-10e5-469e-9da0-c062596c68f6 container/csi-provisioner reason/ContainerExit code/1 cause/Error Lost connection to CSI driver, exiting
#1616680217718296576junit10 days ago
Jan 21 10:16:38.426 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 10:16:38.426 - 16s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 10:16:39.168 - 1s    E disruption/openshift-api connection/reused reason/DisruptionBegan disruption/openshift-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.8:8443: connect: connection refused\n
#1616680217718296576junit10 days ago
Jan 21 09:25:45.520 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:26:03.531 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:26:48.533 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 09:27:24.518 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 10:16:38.426 - 16s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
#1616526754996817920junit10 days ago
Jan 20 23:00:29.599 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:00:29.599 - 8s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:00:30.250 - 999ms E disruption/cache-openshift-api connection/new reason/DisruptionBegan disruption/cache-openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.10:8443: connect: connection refused\n
#1616526754996817920junit10 days ago
Jan 20 23:01:14.528 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:01:14.528 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:01:15.249 - 2s    E disruption/openshift-api connection/new reason/DisruptionBegan disruption/openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.10:8443: connect: connection refused\n
#1616526754996817920junit10 days ago
Jan 20 23:16:59.738 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:16:59.738 - 8s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:17:00.251 - 1s    E disruption/oauth-api connection/reused reason/DisruptionBegan disruption/oauth-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.12:8443: connect: connection refused\n
#1616526754996817920junit10 days ago
Jan 20 23:28:05.577 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:28:05.577 - 18s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:28:06.251 - 1s    E disruption/openshift-api connection/reused reason/DisruptionBegan disruption/openshift-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.10:8443: connect: connection refused\n
#1616526754996817920junit10 days ago
Jan 20 23:28:59.531 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:28:59.531 - 17s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:28:59.583 E ns/e2e-csi-mock-volumes-7348-6572 pod/csi-mockplugin-0 node/ci-op-s6wnz22c-7f65d-bqbcp-worker-centralus2-dwmwx uid/38d519c7-ce10-4cfb-a164-9b39680ab3c5 container/csi-provisioner reason/ContainerExit code/1 cause/Error Lost connection to CSI driver, exiting
#1616526754996817920junit10 days ago
Jan 20 23:32:17.482 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:32:17.482 - 8s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 23:32:22.023 E ns/e2e-volume-3718 pod/nfs-injector node/ci-op-s6wnz22c-7f65d-bqbcp-worker-centralus1-7h4pg uid/b3ea4677-0a19-40c6-91fd-90ba2c309bba container/nfs-injector reason/ContainerExit code/137 cause/Error
#1615800948922257408junit12 days ago
Jan 18 21:53:11.320 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.83:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.83:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.83:8443: connect: no route to host
Jan 18 21:53:11.320 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.83:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.83:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.83:8443: connect: no route to host
Jan 18 21:53:17.817 E ns/openshift-cluster-storage-operator pod/csi-snapshot-webhook-6bb869dfdb-fh7q9 node/ci-op-8xxzrnrx-7f65d-f9svb-master-0 uid/68b608b8-7864-4ce6-9570-530c9e414376 container/webhook reason/ContainerExit code/2 cause/Error
#1615800948922257408junit12 days ago
Jan 18 21:53:11.320 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.83:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.83:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.83:8443: connect: no route to host
periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-gcp-upgrade (all) - 10 runs, 20% failed, 50% of failures match = 10% impact
#1619037969786605568junit3 days ago
Jan 27 19:02:52.534 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 27 19:02:52.534 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 27 19:03:47.937 E ns/openshift-kube-apiserver pod/installer-9-ci-op-vf1syh6v-e823f-trgrk-master-2 node/ci-op-vf1syh6v-e823f-trgrk-master-2 uid/66136ba8-681a-40eb-b73f-897baa52ca54 container/installer reason/ContainerExit code/1 cause/Error ring) (len=10 cap=16) {\n  (string) (len=17) "aggregator-client",\n  (string) (len=30) "localhost-serving-cert-certkey",\n  (string) (len=31) "service-network-serving-certkey",\n  (string) (len=37) "external-loadbalancer-serving-certkey",\n  (string) (len=37) "internal-loadbalancer-serving-certkey",\n  (string) (len=33) "bound-service-account-signing-key",\n  (string) (len=40) "control-plane-node-admin-client-cert-key",\n  (string) (len=31) "check-endpoints-client-cert-key",\n  (string) (len=14) "kubelet-client",\n  (string) (len=16) "node-kubeconfigs"\n },\n OptionalCertSecretNamePrefixes: ([]string) (len=11 cap=16) {\n  (string) (len=17) "user-serving-cert",\n  (string) (len=21) "user-serving-cert-000",\n  (string) (len=21) "user-serving-cert-001",\n  (string) (len=21) "user-serving-cert-002",\n  (string) (len=21) "user-serving-cert-003",\n  (string) (len=21) "user-serving-cert-004",\n  (string) (len=21) "user-serving-cert-005",\n  (string) (len=21) "user-serving-cert-006",\n  (string) (len=21) "user-serving-cert-007",\n  (string) (len=21) "user-serving-cert-008",\n  (string) (len=21) "user-serving-cert-009"\n },\n CertConfigMapNamePrefixes: ([]string) (len=4 cap=4) {\n  (string) (len=20) "aggregator-client-ca",\n  (string) (len=9) "client-ca",\n  (string) (len=29) "control-plane-node-kubeconfig",\n  (string) (len=26) "check-endpoints-kubeconfig"\n },\n OptionalCertConfigMapNamePrefixes: ([]string) (len=1 cap=1) {\n  (string) (len=17) "trusted-ca-bundle"\n },\n CertDir: (string) (len=57) "/etc/kubernetes/static-pod-resources/kube-apiserver-certs",\n ResourceDir: (string) (len=36) "/etc/kubernetes/static-pod-resources",\n PodManifestDir: (string) (len=25) "/etc/kubernetes/manifests",\n Timeout: (time.Duration) 2m0s,\n StaticPodManifestsLockFile: (string) "",\n PodMutationFns: ([]installerpod.PodMutationFunc) <nil>,\n KubeletVersion: (string) ""\n})\nI0127 19:03:22.244551       1 cmd.go:124] Received SIGTERM or SIGINT signal, shutting down the process.\nF0127 19:03:46.237313       1 cmd.go:106] client rate limiter Wait returned an error: context canceled\n
#1619037969786605568junit3 days ago
Jan 27 19:02:52.534 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-openshift-release-master-ci-4.12-e2e-azure-sdn-upgrade (all) - 83 runs, 76% failed, 5% of failures match = 4% impact
#1618934888545652736junit3 days ago
Jan 27 12:46:47.881 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 500, err = an error on the server ("Internal Server Error: \"/apis/template.openshift.io/v1\": Post \"https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": dial tcp 172.30.0.1:443: connect: connection refused") has prevented the request from succeeding
Jan 27 12:46:47.881 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 500, err = an error on the server ("Internal Server Error: \"/apis/template.openshift.io/v1\": Post \"https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": dial tcp 172.30.0.1:443: connect: connection refused") has prevented the request from succeeding
Jan 27 12:47:26.838 E ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-3h94g366-aa502-bc2gh-master-2 node/ci-op-3h94g366-aa502-bc2gh-master-2 uid/eda37ff0-af8d-412e-ad52-458f0665c36d container/kube-apiserver reason/ContainerExit code/2 cause/Error +0x354 fp=0xc007da9c70 sp=0xc007da9b98 pc=0x69d634\ncrypto/tls.(*Conn).Write(0xc01163a588?, {0xc013a1c000, 0x4009, 0x4800})\n	/usr/lib/golang/src/crypto/tls/conn.go:1161 +0x408 fp=0xc007da9d60 sp=0xc007da9c70 pc=0x69e908\nbufio.(*Writer).Write(0xc013e601c0, {0xc013a1c000?, 0xc01163a5f8?, 0x44c393?})\n	/usr/lib/golang/src/bufio/bufio.go:669 +0xed fp=0xc007da9dc0 sp=0xc007da9d60 pc=0x5680ed\ngolang.org/x/net/http2.(*bufferedWriter).Write(0x46c4ae?, {0xc013a1c000?, 0xc007da9e20?, 0x43cee0?})\n	vendor/golang.org/x/net/http2/http2.go:280 +0x151 fp=0xc007da9e08 sp=0xc007da9dc0 pc=0x9744f1\ngolang.org/x/net/http2.(*Framer).endWrite(0xc0137f80e0)\n	vendor/golang.org/x/net/http2/frame.go:371 +0xc5 fp=0xc007da9e60 sp=0xc007da9e08 pc=0x96c665\ngolang.org/x/net/http2.(*Framer).WriteDataPadded(0xc0137f80e0, 0x137e52c0?, 0xc0?, {0xc0151c4000, 0x4000, 0xc0137e52c0?}, {0x0, 0x0, 0xc013a9e600?})\n	vendor/golang.org/x/net/http2/frame.go:694 +0x38a fp=0xc007da9ee0 sp=0xc007da9e60 pc=0x96dd2a\ngolang.org/x/net/http2.(*Framer).WriteData(...)\n	vendor/golang.org/x/net/http2/frame.go:652\ngolang.org/x/net/http2.(*writeData).writeFrame(0xc00a3c0240, {0x5d042a8?, 0xc00b3dcc00?})\n	vendor/golang.org/x/net/http2/write.go:109 +0x53 fp=0xc007da9f30 sp=0xc007da9ee0 pc=0x9952d3\ngolang.org/x/net/http2.(*serverConn).writeFrameAsync(0xc00b3dcc00, {{0x5cddd70?, 0xc00a3c0240?}, 0xc0135d7dd0?, 0x0?})\n	vendor/golang.org/x/net/http2/server.go:815 +0x67 fp=0xc007da9fa8 sp=0xc007da9f30 pc=0x978247\ngolang.org/x/net/http2.(*serverConn).startFrameWrite.func1()\n	vendor/golang.org/x/net/http2/server.go:1221 +0x36 fp=0xc007da9fe0 sp=0xc007da9fa8 pc=0x97adb6\nruntime.goexit()\n	/usr/lib/golang/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc007da9fe8 sp=0xc007da9fe0 pc=0x46e541\ncreated by golang.org/x/net/http2.(*serverConn).startFrameWrite\n	vendor/golang.org/x/net/http2/server.go:1221 +0x36e\nI0127 12:47:26.244180       1 main.go:235] Termination finished with exit code 2\nI0127 12:47:26.244224       1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating"\n
#1618934888545652736junit3 days ago
1 tests failed during this blip (2023-01-27 12:46:44.828870186 +0000 UTC to 2023-01-27 12:46:44.828870186 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
Jan 27 12:46:47.881 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 500, err = an error on the server ("Internal Server Error: \"/apis/template.openshift.io/v1\": Post \"https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": dial tcp 172.30.0.1:443: connect: connection refused") has prevented the request from succeeding
1 tests failed during this blip (2023-01-27 12:46:47.881520654 +0000 UTC to 2023-01-27 12:46:47.881520654 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1618509592919019520junit4 days ago
Jan 26 08:58:37.094 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 08:58:37.094 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 08:58:37.339 - 1s    E disruption/cache-openshift-api connection/new reason/DisruptionBegan disruption/cache-openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.81:8443: connect: connection refused\n
#1618509592919019520junit4 days ago
Jan 26 08:59:04.208 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 08:59:04.208 - 8s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 08:59:04.338 - 1s    E disruption/openshift-api connection/new reason/DisruptionBegan disruption/openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.81:8443: connect: connection refused\n
#1618509592919019520junit4 days ago
Jan 26 09:00:07.091 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 09:00:07.091 - 12s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 09:00:08.339 - 1s    E disruption/openshift-api connection/new reason/DisruptionBegan disruption/openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.81:8443: connect: connection refused\n
#1618509592919019520junit4 days ago
Jan 26 09:00:32.092 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 09:00:32.092 - 2s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 09:00:33.339 - 1s    E disruption/cache-oauth-api connection/new reason/DisruptionBegan disruption/cache-oauth-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.82:8443: connect: connection refused\n
#1618509592919019520junit4 days ago
Jan 26 09:01:07.480 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 09:01:07.480 - 14s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 09:01:08.339 - 1s    E disruption/cache-openshift-api connection/new reason/DisruptionBegan disruption/cache-openshift-api connection/new stopped responding to GET requests over new connections: Get "https://api.ci-op-0jdd2kgv-aa502.ci2.azure.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/default/imagestreams?resourceVersion=0": net/http: timeout awaiting response headers
#1618509592919019520junit4 days ago
Jan 26 09:22:34.013 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 09:22:34.013 - 5s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 09:22:34.340 - 3s    E disruption/openshift-api connection/reused reason/DisruptionBegan disruption/openshift-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.81:8443: connect: connection refused\n
#1615883730583293952junit12 days ago
Jan 19 03:19:20.525 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 19 03:19:20.525 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 19 03:19:50.602 E ns/e2e-check-for-dns-availability-2347 pod/dns-test-94fcc69d-7fcd-4c39-8720-4d3e0669061b-j7xq4 node/ci-op-33x47jd5-aa502-blg6v-worker-centralus1-fvmj4 uid/6dde5fc2-f583-438a-a403-aeec699f79c7 container/querier reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
#1615883730583293952junit12 days ago
Jan 19 03:19:20.525 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-openshift-multiarch-master-nightly-4.10-upgrade-from-nightly-4.9-ocp-remote-libvirt-s390x (all) - 14 runs, 79% failed, 27% of failures match = 21% impact
#1618941940143755264junit3 days ago
Jan 27 14:05:39.804 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.image.openshift.io: not available: failing or missing response from https://10.129.0.10:8443/apis/image.openshift.io/v1: Get "https://10.129.0.10:8443/apis/image.openshift.io/v1": dial tcp 10.129.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.project.openshift.io: not available: failing or missing response from https://10.130.0.13:8443/apis/project.openshift.io/v1: Get "https://10.130.0.13:8443/apis/project.openshift.io/v1": dial tcp 10.130.0.13:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.130.0.13:8443/apis/quota.openshift.io/v1: Get "https://10.130.0.13:8443/apis/quota.openshift.io/v1": dial tcp 10.130.0.13:8443: connect: connection refused
Jan 27 14:05:39.804 - 24s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.image.openshift.io: not available: failing or missing response from https://10.129.0.10:8443/apis/image.openshift.io/v1: Get "https://10.129.0.10:8443/apis/image.openshift.io/v1": dial tcp 10.129.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.project.openshift.io: not available: failing or missing response from https://10.130.0.13:8443/apis/project.openshift.io/v1: Get "https://10.130.0.13:8443/apis/project.openshift.io/v1": dial tcp 10.130.0.13:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.130.0.13:8443/apis/quota.openshift.io/v1: Get "https://10.130.0.13:8443/apis/quota.openshift.io/v1": dial tcp 10.130.0.13:8443: connect: connection refused
Jan 27 14:05:39.994 - 999ms E disruption/oauth-api connection/new disruption/oauth-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.15:8443: connect: connection refused\n
#1618941940143755264junit3 days ago
Jan 27 14:05:39.804 - 24s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.image.openshift.io: not available: failing or missing response from https://10.129.0.10:8443/apis/image.openshift.io/v1: Get "https://10.129.0.10:8443/apis/image.openshift.io/v1": dial tcp 10.129.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.project.openshift.io: not available: failing or missing response from https://10.130.0.13:8443/apis/project.openshift.io/v1: Get "https://10.130.0.13:8443/apis/project.openshift.io/v1": dial tcp 10.130.0.13:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.130.0.13:8443/apis/quota.openshift.io/v1: Get "https://10.130.0.13:8443/apis/quota.openshift.io/v1": dial tcp 10.130.0.13:8443: connect: connection refused
1 tests failed during this blip (2023-01-27 14:05:39.80489188 +0000 UTC to 2023-01-27 14:05:39.80489188 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1617854920289947648junit6 days ago
Jan 24 14:10:15.003 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.128.0.10:8443/apis/apps.openshift.io/v1: Get "https://10.128.0.10:8443/apis/apps.openshift.io/v1": dial tcp 10.128.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/authorization.openshift.io/v1: Get "https://10.130.0.7:8443/apis/authorization.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.build.openshift.io: not available: failing or missing response from https://10.128.0.10:8443/apis/build.openshift.io/v1: Get "https://10.128.0.10:8443/apis/build.openshift.io/v1": dial tcp 10.128.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.security.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/security.openshift.io/v1: Get "https://10.130.0.7:8443/apis/security.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.template.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/template.openshift.io/v1: Get "https://10.130.0.7:8443/apis/template.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused
Jan 24 14:10:15.003 - 95s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.128.0.10:8443/apis/apps.openshift.io/v1: Get "https://10.128.0.10:8443/apis/apps.openshift.io/v1": dial tcp 10.128.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/authorization.openshift.io/v1: Get "https://10.130.0.7:8443/apis/authorization.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.build.openshift.io: not available: failing or missing response from https://10.128.0.10:8443/apis/build.openshift.io/v1: Get "https://10.128.0.10:8443/apis/build.openshift.io/v1": dial tcp 10.128.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.security.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/security.openshift.io/v1: Get "https://10.130.0.7:8443/apis/security.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.template.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/template.openshift.io/v1: Get "https://10.130.0.7:8443/apis/template.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused
Jan 24 14:10:15.519 - 4s    E disruption/openshift-api connection/new disruption/openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: service unavailable\n
#1617854920289947648junit6 days ago
Jan 24 14:10:15.003 - 95s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.128.0.10:8443/apis/apps.openshift.io/v1: Get "https://10.128.0.10:8443/apis/apps.openshift.io/v1": dial tcp 10.128.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/authorization.openshift.io/v1: Get "https://10.130.0.7:8443/apis/authorization.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.build.openshift.io: not available: failing or missing response from https://10.128.0.10:8443/apis/build.openshift.io/v1: Get "https://10.128.0.10:8443/apis/build.openshift.io/v1": dial tcp 10.128.0.10:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.security.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/security.openshift.io/v1: Get "https://10.130.0.7:8443/apis/security.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.template.openshift.io: not available: failing or missing response from https://10.130.0.7:8443/apis/template.openshift.io/v1: Get "https://10.130.0.7:8443/apis/template.openshift.io/v1": dial tcp 10.130.0.7:8443: connect: connection refused
1 tests failed during this blip (2023-01-24 14:10:15.003211412 +0000 UTC to 2023-01-24 14:10:15.003211412 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
#1616405466064097280junit10 days ago
Jan 20 14:20:08.154 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 14:20:08.154 - 32s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 20 14:20:08.377 - 1s    E disruption/openshift-api connection/new disruption/openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: service unavailable\n
#1616405466064097280junit10 days ago
Jan 20 14:20:08.154 - 32s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
1 tests failed during this blip (2023-01-20 14:20:08.154061097 +0000 UTC to 2023-01-20 14:20:08.154061097 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
periodic-ci-openshift-release-master-ci-4.11-e2e-azure-upgrade (all) - 10 runs, 30% failed, 33% of failures match = 10% impact
#1618883813121724416junit4 days ago
Jan 27 09:53:41.339 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 27 09:53:41.339 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 27 09:54:28.218 E ns/e2e-check-for-dns-availability-9249 pod/dns-test-52f5a685-53bd-4212-828f-37a0dc53d186-4xj7g node/ci-op-fy1dcm8q-5eb8a-9nqdp-worker-centralus3-9zwbj uid/2b882945-8bb9-4b3e-aca7-d252e47e4077 container/querier reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
#1618883813121724416junit4 days ago
Jan 27 09:53:41.339 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-sdn-upgrade (all) - 84 runs, 49% failed, 2% of failures match = 1% impact
#1618871985046032384junit4 days ago
Jan 27 08:53:21.007 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.104:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.104:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.104:8443: connect: no route to host
Jan 27 08:53:21.007 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.104:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.104:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.104:8443: connect: no route to host
Jan 27 08:53:28.806 E ns/openshift-cluster-csi-drivers pod/gcp-pd-csi-driver-operator-7b9c9d444-dd8xv node/ci-op-3pzlts61-4f6ab-nmjps-master-1 uid/b3479347-8432-4ff5-a57b-e4f383989976 container/gcp-pd-csi-driver-operator reason/ContainerExit code/1 cause/Error
#1618871985046032384junit4 days ago
Jan 27 08:53:21.007 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.104:8443/apis/apps.openshift.io/v1: Get "https://10.129.0.104:8443/apis/apps.openshift.io/v1": dial tcp 10.129.0.104:8443: connect: no route to host
periodic-ci-openshift-release-master-ci-4.9-e2e-gcp-upgrade (all) - 13 runs, 54% failed, 14% of failures match = 8% impact
#1618730514070376448junit4 days ago
Jan 26 23:33:55.407 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 23:33:55.407 - 172ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 23:34:33.000 - 10s   E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests on reused connections
#1618730514070376448junit4 days ago
Jan 26 23:33:55.407 - 172ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-sdn-upgrade (all) - 319 runs, 62% failed, 2% of failures match = 1% impact
#1618730775140634624junit4 days ago
Jan 26 22:55:38.350 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 22:55:38.350 - 4s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 26 22:56:01.053 E ns/openshift-machine-api pod/control-plane-machine-set-operator-7dddf9666b-d7t78 node/ip-10-0-248-216.us-west-2.compute.internal uid/b7f70cd2-93a5-47e8-a2ed-3ee0df3aa778 container/control-plane-machine-set-operator reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
#1618730775140634624junit4 days ago
Jan 26 22:55:38.350 - 4s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
#1618101825800507392junit6 days ago
Jan 25 05:30:09.299 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 25 05:30:09.299 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 25 05:31:26.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://ac784f0012a654dbb9c89b81d507225f-1339146265.us-west-1.elb.amazonaws.com:80/echo?msg=Hello": net/http: timeout awaiting response headers
#1618101825800507392junit6 days ago
Jan 25 05:30:09.299 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
#1616881549708365824junit9 days ago
Jan 21 20:20:04.131 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 21 20:20:04.131 - 623ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 21 20:20:04.174 E clusteroperator/authentication condition/Available status/False reason/APIServerDeployment_NoDeployment changed: APIServerDeploymentAvailable: deployment/openshift-oauth-apiserver: could not be retrieved
#1616881549708365824junit9 days ago
Jan 21 20:20:04.131 - 623ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-azure-sdn-upgrade (all) - 531 runs, 73% failed, 2% of failures match = 1% impact
#1618599003396509696junit4 days ago
Jan 26 14:22:50.039 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Internal desc = server closed the stream without sending trailers
Jan 26 14:22:50.039 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Internal desc = server closed the stream without sending trailers
Jan 26 14:32:08.548 E ns/openshift-kube-scheduler-operator pod/openshift-kube-scheduler-operator-7db565db9-gvjkr node/ci-op-jcrhw4hk-7cadb-8cgx5-master-1 uid/238438d4-9cae-4b5d-904c-7fdc9d5a4288 container/kube-scheduler-operator-container reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
#1618599003396509696junit4 days ago
Jan 26 14:22:50.039 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Internal desc = server closed the stream without sending trailers
#1617510441137737728junit7 days ago
Jan 23 15:10:20.257 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 23 15:10:20.257 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 23 15:10:58.000 - 1s    E disruption/service-load-balancer-with-pdb connection/new reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/new stopped responding to GET requests over new connections: Get "http://20.9.104.240:80/echo?msg=Hello": dial tcp 20.9.104.240:80: i/o timeout
#1617510441137737728junit7 days ago
Jan 23 15:10:20.257 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
#1617040678796136448junit9 days ago
Jan 22 07:21:36.228 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:21:36.228 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:21:36.767 - 999ms E disruption/openshift-api connection/new reason/DisruptionBegan disruption/openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.44:8443: connect: connection refused\n
#1617040678796136448junit9 days ago
Jan 22 07:38:20.470 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:38:20.470 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:38:21.767 - 1s    E disruption/cache-openshift-api connection/reused reason/DisruptionBegan disruption/cache-openshift-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.44:8443: connect: connection refused\n
#1617040678796136448junit9 days ago
Jan 22 07:38:33.225 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:38:33.225 - 18s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:38:33.767 - 2s    E disruption/openshift-api connection/reused reason/DisruptionBegan disruption/openshift-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.44:8443: connect: connection refused\n
#1617040678796136448junit9 days ago
Jan 22 07:39:27.222 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:39:27.222 - 29s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:39:27.768 - 999ms E disruption/cache-openshift-api connection/new reason/DisruptionBegan disruption/cache-openshift-api connection/new stopped responding to GET requests over new connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.44:8443: connect: connection refused\n
#1617040678796136448junit9 days ago
Jan 22 08:28:15.516 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 08:28:15.516 - 26s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 08:28:17.767 - 2s    E disruption/openshift-api connection/reused reason/DisruptionBegan disruption/openshift-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.130.0.98:8443: connect: connection refused\n
#1617040678796136448junit9 days ago
Jan 22 07:21:36.228 - 9s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:38:20.470 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:38:33.225 - 18s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 07:39:27.222 - 29s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 22 08:28:15.516 - 26s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
#1616315420707393536junit11 days ago
Jan 20 07:49:41.459 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 20 07:49:41.459 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 20 07:50:44.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://20.98.184.123:80/echo?msg=Hello": net/http: timeout awaiting response headers
#1616315420707393536junit11 days ago
Jan 20 07:49:41.459 - 1s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
#1616042712828481536junit11 days ago
Jan 19 14:14:56.337 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 19 14:14:56.337 - 5s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 19 14:14:56.769 E clusteroperator/authentication condition/Available status/False reason/OAuthServerRouteEndpointAccessibleController_EndpointUnavailable changed: OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.ci-op-rq2t19gj-7cadb.ci2.azure.devcluster.openshift.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
#1616042712828481536junit11 days ago
Jan 19 14:14:56.337 - 5s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
8 tests failed during this blip (2023-01-19 14:14:56.337632033 +0000 UTC to 2023-01-19 14:14:56.337632033 +0000 UTC): [sig-node] Probing container should *not* be restarted by liveness probe because startup probe delays it [Suite:openshift/conformance/parallel] [Suite:k8s]
#1615672483661221888junit12 days ago
Jan 18 13:29:50.146 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 18 13:29:50.146 - 626ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 18 13:30:14.321 - 1s    E disruption/oauth-api connection/reused reason/DisruptionBegan disruption/oauth-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ci-op-wqgswys6-7cadb.ci2.azure.devcluster.openshift.com:6443/apis/oauth.openshift.io/v1/oauthclients": net/http: timeout awaiting response headers
#1615672483661221888junit12 days ago
Jan 18 13:29:50.146 - 626ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-openshift-release-master-ci-4.10-e2e-gcp-upgrade (all) - 59 runs, 39% failed, 9% of failures match = 3% impact
#1618484992894046208junit5 days ago
Jan 26 07:09:42.263 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 26 07:09:42.263 - 758ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 26 07:09:57.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://34.172.129.162:80/echo?msg=Hello": net/http: timeout awaiting response headers
#1618484992894046208junit5 days ago
Jan 26 07:09:42.263 - 758ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
#1615869303553265664junit12 days ago
Jan 19 01:57:52.396 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 19 01:57:52.396 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 19 01:58:34.263 E ns/openshift-dns pod/node-resolver-rld2m node/ci-op-78pm2xx5-a9175-cxh7r-worker-a-87bcs container/dns-node-resolver reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
#1615869303553265664junit12 days ago
Jan 19 01:57:52.396 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.10-upgrade-from-stable-4.9-e2e-openstack-upgrade (all) - 2 runs, 100% failed, 50% of failures match = 50% impact
#1616918158495453184junit9 days ago
Jan 21 23:39:43.883 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 23:39:43.883 - 785ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 23:39:48.693 E ns/openshift-sdn pod/sdn-controller-56c7p node/0mrc40mp-c88ed-rwcmj-master-1 container/sdn-controller reason/ContainerExit code/2 cause/Error 2366       1 vnids.go:116] Allocated netid 15559912 for namespace "e2e-oauth-api-new-3761"\nI0121 22:49:03.216993       1 vnids.go:116] Allocated netid 5587988 for namespace "e2e-openshift-api-reused-1646"\nI0121 22:49:03.339857       1 vnids.go:116] Allocated netid 11377374 for namespace "e2e-image-registry-new-7529"\nI0121 22:49:03.421184       1 vnids.go:116] Allocated netid 13387307 for namespace "e2e-kube-api-reused-2176"\nI0121 22:49:03.640987       1 vnids.go:116] Allocated netid 14805775 for namespace "e2e-k8s-sig-storage-sig-api-machinery-configmap-upgrade-9571"\nI0121 22:49:03.869151       1 vnids.go:116] Allocated netid 7470485 for namespace "e2e-image-registry-reused-3552"\nI0121 22:49:04.114548       1 vnids.go:116] Allocated netid 1688479 for namespace "e2e-k8s-service-load-balancer-with-pdb-reused-3953"\nI0121 22:49:04.220294       1 vnids.go:116] Allocated netid 10207044 for namespace "e2e-oauth-api-reused-128"\nI0121 22:49:04.421804       1 vnids.go:116] Allocated netid 1587163 for namespace "e2e-k8s-sig-apps-replicaset-upgrade-7956"\nI0121 22:49:04.708781       1 vnids.go:116] Allocated netid 5756392 for namespace "e2e-ingress-to-console-new-2714"\nI0121 22:49:04.826255       1 vnids.go:116] Allocated netid 11994880 for namespace "e2e-ingress-to-console-reused-4491"\nI0121 22:49:05.018754       1 vnids.go:116] Allocated netid 7348374 for namespace "e2e-k8s-sig-apps-daemonset-upgrade-5604"\nI0121 22:49:05.255847       1 vnids.go:116] Allocated netid 977962 for namespace "e2e-prometheus-metrics-available-after-upgrade-76"\nI0121 22:49:05.413139       1 vnids.go:116] Allocated netid 7436279 for namespace "e2e-ingress-to-oauth-server-new-6154"\nI0121 22:49:05.617407       1 vnids.go:116] Allocated netid 9572309 for namespace "e2e-k8s-sig-storage-sig-api-machinery-secret-upgrade-9669"\nI0121 23:31:12.981812       1 vnids.go:116] Allocated netid 11308932 for namespace "openshift-cloud-network-config-controller"\nE0121 23:36:14.199983       1 leaderelection.go:367] Failed to update lock: etcdserver: request timed out\n
#1616918158495453184junit9 days ago
Jan 22 00:15:59.263 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "route.openshift.io.v1" is not ready: [an attempt failed with statusCode = 500, err = an error on the server ("Internal Server Error: \"/apis/route.openshift.io/v1\": Post \"https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": read tcp 10.129.0.13:45582-&gt;172.30.0.1:443: read: connection timed out") has prevented the request from succeeding, an attempt failed with statusCode = 503, err = the server is currently unable to handle the request]
Jan 22 00:15:59.263 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "route.openshift.io.v1" is not ready: [an attempt failed with statusCode = 500, err = an error on the server ("Internal Server Error: \"/apis/route.openshift.io/v1\": Post \"https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": read tcp 10.129.0.13:45582-&gt;172.30.0.1:443: read: connection timed out") has prevented the request from succeeding, an attempt failed with statusCode = 503, err = the server is currently unable to handle the request]
Jan 22 00:16:40.186 - 29s   E alert/etcdGRPCRequestsSlow node/10.0.3.55:9979 ns/openshift-etcd pod/etcd-0mrc40mp-c88ed-rwcmj-master-0 ALERTS{alertname="etcdGRPCRequestsSlow", alertstate="firing", endpoint="etcd-metrics", grpc_method="Range", grpc_service="etcdserverpb.KV", instance="10.0.3.55:9979", job="etcd", namespace="openshift-etcd", pod="etcd-0mrc40mp-c88ed-rwcmj-master-0", prometheus="openshift-monitoring/k8s", service="etcd", severity="critical"}
#1616918158495453184junit9 days ago
Jan 21 23:39:43.883 - 785ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
1 tests failed during this blip (2023-01-21 23:39:43.883009829 +0000 UTC to 2023-01-21 23:39:43.883009829 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
Jan 22 00:15:59.263 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request\nAPIServicesAvailable: "route.openshift.io.v1" is not ready: [an attempt failed with statusCode = 500, err = an error on the server ("Internal Server Error: \"/apis/route.openshift.io/v1\": Post \"https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": read tcp 10.129.0.13:45582-&gt;172.30.0.1:443: read: connection timed out") has prevented the request from succeeding, an attempt failed with statusCode = 503, err = the server is currently unable to handle the request]
1 tests failed during this blip (2023-01-22 00:15:59.263237268 +0000 UTC to 2023-01-22 00:15:59.263237268 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-gcp-upgrade (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1616837123720089600junit9 days ago
Jan 21 18:11:18.190 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 18:11:18.190 - 209ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 21 18:11:35.900 E ns/openshift-multus pod/multus-additional-cni-plugins-l2hl9 node/ci-op-s4tt0r80-ff28f-rnl5h-master-2 container/kube-multus-additional-cni-plugins reason/ContainerExit code/143 cause/Error
#1616837123720089600junit9 days ago
Jan 21 18:11:18.190 - 209ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.11-upgrade-from-stable-4.10-e2e-openstack-upgrade (all) - 6 runs, 100% failed, 17% of failures match = 17% impact
#1616612894948986880junit10 days ago
Jan 21 03:58:35.328 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 21 03:58:35.328 - 204ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 21 03:58:35.749 - 2s    E disruption/cache-oauth-api connection/reused reason/DisruptionBegan disruption/cache-oauth-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: error trying to reach service: dial tcp 10.129.0.14:8443: connect: connection refused\n
#1616612894948986880junit10 days ago
Jan 21 03:58:35.328 - 204ms E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
1 tests failed during this blip (2023-01-21 03:58:35.328940835 +0000 UTC to 2023-01-21 03:58:35.328940835 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
periodic-ci-openshift-multiarch-master-nightly-4.11-upgrade-from-nightly-4.10-ocp-remote-libvirt-s390x (all) - 10 runs, 80% failed, 13% of failures match = 10% impact
#1616178950164189184junit11 days ago
Jan 19 22:41:37.114 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 19 22:41:37.114 - 41s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
Jan 19 22:41:37.221 - 999ms E disruption/cache-openshift-api connection/reused reason/DisruptionBegan disruption/cache-openshift-api connection/reused stopped responding to GET requests over reused connections: error running request: 503 Service Unavailable: service unavailable\n
#1616178950164189184junit11 days ago
Jan 19 22:42:20.018 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.129.0.85:8443/apis/quota.openshift.io/v1: Get "https://10.129.0.85:8443/apis/quota.openshift.io/v1": dial tcp 10.129.0.85:8443: connect: connection refused
Jan 19 22:42:20.018 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.129.0.85:8443/apis/quota.openshift.io/v1: Get "https://10.129.0.85:8443/apis/quota.openshift.io/v1": dial tcp 10.129.0.85:8443: connect: connection refused
Jan 19 22:42:20.033 E clusteroperator/operator-lifecycle-manager-packageserver condition/Available status/False reason/ClusterServiceVersionNotSucceeded changed: ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: ComponentUnhealthy, message: apiServices not installed
#1616178950164189184junit11 days ago
Jan 19 22:54:16.334 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/apps.openshift.io/v1: Get "https://10.128.0.90:8443/apis/apps.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/authorization.openshift.io/v1: Get "https://10.128.0.90:8443/apis/authorization.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.image.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/image.openshift.io/v1: Get "https://10.128.0.90:8443/apis/image.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.project.openshift.io: not available: failing or missing response from https://10.130.0.9:8443/apis/project.openshift.io/v1: Get "https://10.130.0.9:8443/apis/project.openshift.io/v1": dial tcp 10.130.0.9:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/quota.openshift.io/v1: Get "https://10.128.0.90:8443/apis/quota.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.route.openshift.io: not available: failing or missing response from https://10.130.0.9:8443/apis/route.openshift.io/v1: Get "https://10.130.0.9:8443/apis/route.openshift.io/v1": dial tcp 10.130.0.9:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.template.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/template.openshift.io/v1: Get "https://10.128.0.90:8443/apis/template.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused
Jan 19 22:54:16.334 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/apps.openshift.io/v1: Get "https://10.128.0.90:8443/apis/apps.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/authorization.openshift.io/v1: Get "https://10.128.0.90:8443/apis/authorization.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.image.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/image.openshift.io/v1: Get "https://10.128.0.90:8443/apis/image.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.project.openshift.io: not available: failing or missing response from https://10.130.0.9:8443/apis/project.openshift.io/v1: Get "https://10.130.0.9:8443/apis/project.openshift.io/v1": dial tcp 10.130.0.9:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/quota.openshift.io/v1: Get "https://10.128.0.90:8443/apis/quota.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.route.openshift.io: not available: failing or missing response from https://10.130.0.9:8443/apis/route.openshift.io/v1: Get "https://10.130.0.9:8443/apis/route.openshift.io/v1": dial tcp 10.130.0.9:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.template.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/template.openshift.io/v1: Get "https://10.128.0.90:8443/apis/template.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused
Jan 19 22:54:16.438 E clusteroperator/operator-lifecycle-manager-packageserver condition/Available status/False reason/ClusterServiceVersionNotSucceeded changed: ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: ComponentUnhealthy, message: apiServices not installed
#1616178950164189184junit11 days ago
Jan 19 22:41:37.114 - 41s   E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
1 tests failed during this blip (2023-01-19 22:41:37.114453561 +0000 UTC to 2023-01-19 22:41:37.114453561 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
Jan 19 22:42:20.018 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.129.0.85:8443/apis/quota.openshift.io/v1: Get "https://10.129.0.85:8443/apis/quota.openshift.io/v1": dial tcp 10.129.0.85:8443: connect: connection refused
1 tests failed during this blip (2023-01-19 22:42:20.018790868 +0000 UTC to 2023-01-19 22:42:20.018790868 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
Jan 19 22:54:16.334 - 3s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/apps.openshift.io/v1: Get "https://10.128.0.90:8443/apis/apps.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.authorization.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/authorization.openshift.io/v1: Get "https://10.128.0.90:8443/apis/authorization.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.image.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/image.openshift.io/v1: Get "https://10.128.0.90:8443/apis/image.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.project.openshift.io: not available: failing or missing response from https://10.130.0.9:8443/apis/project.openshift.io/v1: Get "https://10.130.0.9:8443/apis/project.openshift.io/v1": dial tcp 10.130.0.9:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.quota.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/quota.openshift.io/v1: Get "https://10.128.0.90:8443/apis/quota.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.route.openshift.io: not available: failing or missing response from https://10.130.0.9:8443/apis/route.openshift.io/v1: Get "https://10.130.0.9:8443/apis/route.openshift.io/v1": dial tcp 10.130.0.9:8443: connect: connection refused\nAPIServicesAvailable: apiservices.apiregistration.k8s.io/v1.template.openshift.io: not available: failing or missing response from https://10.128.0.90:8443/apis/template.openshift.io/v1: Get "https://10.128.0.90:8443/apis/template.openshift.io/v1": dial tcp 10.128.0.90:8443: connect: connection refused
1 tests failed during this blip (2023-01-19 22:54:16.334268518 +0000 UTC to 2023-01-19 22:54:16.334268518 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
periodic-ci-openshift-release-master-nightly-4.12-upgrade-from-stable-4.10-e2e-aws-sdn-upgrade-paused (all) - 5 runs, 80% failed, 25% of failures match = 20% impact
#1615394752478318592junit13 days ago
Jan 17 18:53:45.374 E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServices_Error changed: APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 17 18:53:45.374 - 7s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Jan 17 18:54:19.479 E ns/openshift-etcd pod/etcd-ip-10-0-135-81.ec2.internal node/ip-10-0-135-81.ec2.internal uid/46628bff-d427-4e2d-a429-cef2472fae40 invariant violation: container statuses were removed
#1615394752478318592junit13 days ago
Jan 17 18:53:45.374 - 7s    E clusteroperator/openshift-apiserver condition/Available status/False reason/APIServicesAvailable: rpc error: code = Unknown desc = malformed header: missing HTTP content-type

Found in 1.40% of runs (2.36% of failures) across 3289 total runs and 152 jobs (59.35% failed) in 25.305s - clear search | chart view - source code located on github