Issue 15722416: [release-4.15] Install failure for console operator
Description: Description of problem:
 
 Reviewing [4.15 Install failures|https://sippy.dptools.openshift.org/sippy-ng/jobs/4.15/runs?filters=%257B%2522items%2522%253A%255B%257B%2522columnField%2522%253A%2522failed_test_names%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522install%2520should%2520succeed%253A%2520overall%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522upgrade-minor%2522%252C%2522not%2522%253Atrue%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522aggregated%2522%252C%2522not%2522%253Atrue%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522never-stable%2522%252C%2522not%2522%253Atrue%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522single-node%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522heterogeneous%2522%257D%255D%252C%2522linkOperator%2522%253A%2522and%2522%257D&sort=desc&sortField=timestamp] there are a number of variants impacted by recent install failures.
 
 [search.ci: Cluster operator console is not available|https://search.ci.openshift.org/?search=Cluster+operator+console+is+not+available&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=4.15&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job]
 
 Jobs like [periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-serial|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-serial/1732897807150354432] show failures that appear to start with [4.15.0-0.nightly-2023-12-07-225558|https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-serial/1732897807150354432/build-log.txt] have installation failures due to [console-operator|https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-serial/1732897807150354432/artifacts/e2e-gcp-sdn-serial/gather-extra/artifacts/pods/openshift-console-operator_console-operator-8697d569d7-z7vq9_console-operator.log]
 {code:none}
 ConsoleOperator reconciliation failed: Operation cannot be fulfilled on consoles.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
 {code}
  
 
  
 
 [4.15.0-0.nightly-2023-12-07-225558|https://sippy.dptools.openshift.org/sippy-ng/release/4.15/tags/4.15.0-0.nightly-2023-12-07-225558/pull_requests?pageSize=50&sort=desc&sortField=name] contains [console-operator/pull/814|https://github.com/openshift/console-operator/pull/814], noting in case it is related
 
  
 
  
 
 Version-Release number of selected component (if applicable):
 {code:none}
  4.15   {code}
 How reproducible:
 {code:none}
     {code}
 Steps to Reproduce:
 {code:none}
     1. Review link to install failures above
     2.
     3.
     {code}
 Actual results:
 {code:none}
     {code}
 Expected results:
 {code:none}
     {code}
 Additional info:
 [periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-sdn|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-sdn/1735541698454360064]
 [periodic-ci-openshift-release-master-nightly-4.15-e2e-aws-sdn-upgrade|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-aws-sdn-upgrade/1736354213404872704]
 [periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-ovn-upgrade|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-ovn-upgrade/1733023780692824064]
Status: ASSIGNED

Issue 15681865: Install failure for console operator
Description: Description of problem:
 
 Reviewing [4.15 Install failures|https://sippy.dptools.openshift.org/sippy-ng/jobs/4.15/runs?filters=%257B%2522items%2522%253A%255B%257B%2522columnField%2522%253A%2522failed_test_names%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522install%2520should%2520succeed%253A%2520overall%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522upgrade-minor%2522%252C%2522not%2522%253Atrue%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522aggregated%2522%252C%2522not%2522%253Atrue%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522never-stable%2522%252C%2522not%2522%253Atrue%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522single-node%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522heterogeneous%2522%257D%255D%252C%2522linkOperator%2522%253A%2522and%2522%257D&sort=desc&sortField=timestamp] (install should succeed: overall) there are a number of variants impacted by recent install failures.
 
 [search.ci: Cluster operator console is not available|https://search.ci.openshift.org/?search=Cluster+operator+console+is+not+available&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=4.15&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job]
 
 Jobs like [periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-serial|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-serial/1732897807150354432] show failures that appear to start with [4.15.0-0.nightly-2023-12-07-225558|https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-serial/1732897807150354432/build-log.txt] have installation failures due to [console-operator|https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-serial/1732897807150354432/artifacts/e2e-gcp-sdn-serial/gather-extra/artifacts/pods/openshift-console-operator_console-operator-8697d569d7-z7vq9_console-operator.log]
 {code:none}
 ConsoleOperator reconciliation failed: Operation cannot be fulfilled on consoles.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
 {code}
  
 
  
 
 [4.15.0-0.nightly-2023-12-07-225558|https://sippy.dptools.openshift.org/sippy-ng/release/4.15/tags/4.15.0-0.nightly-2023-12-07-225558/pull_requests?pageSize=50&sort=desc&sortField=name] contains [console-operator/pull/814|https://github.com/openshift/console-operator/pull/814], noting in case it is related
 
  
 
  
 
 Version-Release number of selected component (if applicable):
 {code:none}
  4.15   {code}
 How reproducible:
 {code:none}
     {code}
 Steps to Reproduce:
 {code:none}
     1. Review link to install failures above
     2.
     3.
     {code}
 Actual results:
 {code:none}
     {code}
 Expected results:
 {code:none}
     {code}
 Additional info:
 [periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-sdn|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-sdn/1735541698454360064]
 [periodic-ci-openshift-release-master-nightly-4.15-e2e-aws-sdn-upgrade|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-aws-sdn-upgrade/1736354213404872704]
 [periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-ovn-upgrade|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-ovn-upgrade/1733023780692824064]
Status: Verified
Comment 23840324 by YaDan Pei at 2024-01-22T06:27:47.920+0000
[Search Cluster operator console is not available in 4.16 jobs for latest 2 days|https://search.ci.openshift.org/?search=Cluster+operator+console+is+not+available&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=4.16&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job] we can still see many occurrences of console installation failure 
Comment 23887457 by YaDan Pei at 2024-01-26T02:57:39.752+0000
[Search Cluster operator console is not available in 2 days|https://search.ci.openshift.org/?search=Cluster+operator+console+is+not+available&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=4.16&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job] there are 4 failures, these failures occur during 4.15 installation in an upgrade job