Operator regenerates DB password without redeploying Postgres

[metlos]

Describe the bug

If a user changes the tag of the che image in the CR, the server is redeployed with that new image.

But the operator also creates a new random DB password and updates the che config map with it, without redeploying the Postgres server, which leads to a non-functioning installation.

Che version

N/A

Steps to reproduce

1. build and install the operator into some namespace
2. deploy a che cluster CR without specifying any che image or image tag, letting the operator deploy the default version
3. Update the CR with a specific image tag (e.g. set image tag to nightly for example)
4. Apply your changed CR to the cluster
5. Wait for the operator to pick up changes and redeploy the che server
6. Notice the the discrepancy between POSTGRESQL_PASSWORD env var of the postgres pod and the CHE_JDBC_PASSWORD env var in the che config map
7. Witness the redeployment of the Che server fail.

Expected behavior

The db password should not change because of the changed che server image.

Runtime

minikube

$ minikube version  
minikube version: v1.3.1
commit: ca60a424ce69a4d79f502650199ca2b52f29e631
$ kubectl version                                    
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Installation method

che-operator from master

Environment

Linux

[ibuziuk]

@davidfestal could you please take a look and comment ?

[davidfestal]

I'm looking into it.

[metlos]

To give more detail, here is exactly what I'm doing in terms of the commands executed. This assumes che-operator master branch checked out and be present in the root of the checkout:

eval $(minikube docker-env)
kubectl create namespace op-test
docker build -t che/operator .
kubectl apply -f deploy/service_account.yaml -n=op-test
kubectl apply -f deploy/role.yaml -n=op-test
kubectl apply -f deploy/role_binding.yaml -n=op-test
kubectl apply -f deploy/crds/org_v1_che_crd.yaml -n=op-test
kubectl apply -f deploy/operator-local.yaml -n op-test
cat <<EOF > cr.yaml
#
#  Copyright (c) 2012-2019 Red Hat, Inc.
#    This program and the accompanying materials are made
#    available under the terms of the Eclipse Public License 2.0
#    which is available at https://www.eclipse.org/legal/epl-2.0/
#
#  SPDX-License-Identifier: EPL-2.0
#
#  Contributors:
#    Red Hat, Inc. - initial API and implementation

apiVersion: org.eclipse.che/v1
kind: CheCluster
metadata:
  name: eclipse-che
spec:
  server:
    # server image used in Che deployment
    cheImage: ''
    # tag of an image used in Che deployment
    cheImageTag: ''
    # image:tag used in Devfile registry deployment
    devfileRegistryImage: ''
    # image:tag used in plugin registry deployment
    pluginRegistryImage: ''
    # defaults to `che`. When set to `codeready`, CodeReady Workspaces is deployed
    # the difference is in images, labels, exec commands
    cheFlavor: ''
    # specifies a custom cluster role to user for the Che workspaces
    # Uses the default roles if left blank.
    cheWorkspaceClusterRole: ''
    # when set to true the operator will attempt to get a secret in OpenShift router namespace
    # to add it to Java trust store of Che server. Requires cluster-admin privileges for operator service account
    selfSignedCert: false
    # TLS mode for Che. Make sure you either have public cert, or set selfSignedCert to true
    tlsSupport: false
    # protocol+hostname of a proxy server. Automatically added as JAVA_OPTS and https(s)_proxy
    # to Che server and workspaces containers
    proxyURL: ''
    # port of a proxy server
    proxyPort: ''
    # username for a proxy server
    proxyUser: ''
    # password for a proxy user
    proxyPassword: ''
    # a list of non-proxy hosts. Use | as delimiter, eg localhost|my.host.com|123.42.12.32
    nonProxyHosts: ''
    # sets mem request for server deployment. Defaults to 512Mi
    serverMemoryRequest: ''
    # sets mem limit for server deployment. Defaults to 1Gi
    serverMemoryLimit: ''
  database:
    # when set to true, the operator skips deploying Postgres, and passes connection details of existing DB to Che server
    # otherwise a Postgres deployment is created
    externalDb: false
    # Postgres Database hostname that Che server uses to connect to. Defaults to postgres
    chePostgresHostName: ''
    # Postgres Database port that Che server uses to connect to. Defaults to 5432
    chePostgresPort: ''
    # Postgres user that Che server when making a db connection. Defaults to pgche
    chePostgresUser: ''
    # password of a postgres user. Auto-generated when left blank
    chePostgresPassword: ''
    # Postgres database name that Che server uses to connect to. Defaults to dbche
    chePostgresDb: ''
    # Postgres deployment in format image:tag. Defaults to registry.redhat.io/rhscl/postgresql-96-rhel7 (see pkg/deploy/defaults.go for latest tag)
    postgresImage: ''
  storage:
    # persistent volume claim strategy for Che server. Can be common (all workspaces PVCs in one volume),
    # per-workspace (one PVC per workspace for all declared volumes) and unique (one PVC per declared volume). Defaults to common
    pvcStrategy: 'per-workspace'
    # size of a persistent volume claim for workspaces. Defaults to 1Gi
    pvcClaimSize: '1Gi'
    # instruct Che server to launch a special pod to precreate a subpath in a PV
    preCreateSubPaths: true
    # image:tag for preCreateSubPaths jobs
    pvcJobsImage: ''
    # keep blank unless you need to use a non default storage class for Postgres PVC
    postgresPVCStorageClassName: ''
    # keep blank unless you need to use a non default storage class for workspace PVC(s)
    workspacePVCStorageClassName: ''

  auth:
    # instructs operator on whether or not to deploy Keycloak/RH SSO instance. When set to true provision connection details
    externalIdentityProvider: false
    # retrieved from respective route/ingress unless explicitly specified in CR (when ExternalKeycloak is true)
    identityProviderURL: ''
    # password for keycloak database user. Auto generated if left blank
    keycloakPostgresPassword: ''
    # desired admin username of Keycloak admin user (applicable only when externalIdentityProvider is false)
    identityProviderAdminUserName: ''
    # desired password of Keycloak admin user (applicable only when externalIdentityProvider is false)
    identityProviderPassword: 'admin'
    # name of a keycloak realm. This realm will be created, when externalIdentityProvider is true, otherwise passed to Che server
    identityProviderRealm: ''
    # id of a keycloak client. This client will be created, when externalIdentityProvider is false, otherwise passed to Che server
    identityProviderClientId: ''
    # instructs an Operator to enable OpenShift v3 identity provider in Keycloak,
    # as well as create respective oAuthClient and configure Che configMap accordingly
    openShiftoAuth: false
    # name of oAuthClient used in OpenShift v3 identity provider in Keycloak realm. Auto generated if left blank
    oAuthClientName: ''
    # secret used in oAuthClient. Auto generated if left blank
    oAuthSecret: ''
    # image:tag used in Keycloak deployment
    identityProviderImage: ''
  k8s:
    # your global ingress domain
    ingressDomain: '192.168.99.127.nip.io'
    # kubernetes.io/ingress.class, defaults to nginx
    ingressClass: ''
    # IngressStrategy is the way ingresses are created.
    # Can be multi-host (host is explicitly provided in ingress, <ingress-name>-<namespace>.<global-ingress-domain>),
    # single-host (host is provided, path based rules, <ingress-domain>/path) and default-host *(no host is provided, path based rules)
    ingressStrategy: ''
    # secret name used for tls termination
    tlsSecretName: ''
    # FSGroup the Che POD and Workspace pod containers should run in  
    securityContextFsGroup: '' 
    # User the Che POD and Workspace pod containers should run as  
    securityContextRunAsUser: '' 
EOF

kubectl apply -f cr.yaml -n=op-test

### wait until the Che pod is ready

sed -i "s/cheImageTag: ''/cheImageTag: 'nightly'/" cr.yaml

kubectl apply -f cr.yaml -n=op-test

and here are the operator logs for the time:

$ kubectl logs che-operator-858ff74779-wjnrs -n op-test 
time="2019-09-24T10:13:31Z" level=info msg="Default 'info' log level is applied" 
time="2019-09-24T10:13:31Z" level=info msg="Go Version: go1.11.13" 
time="2019-09-24T10:13:31Z" level=info msg="Go OS/Arch: linux/amd64" 
time="2019-09-24T10:13:31Z" level=info msg="operator-sdk Version: v0.5.0" 
time="2019-09-24T10:13:31Z" level=info msg="Operator is running on Kubernetes" 
time="2019-09-24T10:13:32Z" level=info msg="Registering Che Components Types" 
time="2019-09-24T10:13:32Z" level=info msg="Starting the Cmd" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object: ServiceAccount, name: che" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object: ServiceAccount, name: che-workspace" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object: Role, name: exec" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object: Role, name: view" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object: RoleBinding, name: che" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object: RoleBinding, name: che-workspace-exec" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object: RoleBinding, name: che-workspace-view" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with auto-generated CheCluster DB password: password-hidden" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=error msg="Failed to get keycloak deployment: Deployment.apps "keycloak" not found" 
time="2019-09-24T10:14:05Z" level=info msg="Disregard the error. No existing Identity provider deployment found. Generating passwd" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with auto-generated Keycloak DB password: password-hidden" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=error msg="Failed to get keycloak deployment: Deployment.apps "keycloak" not found" 
time="2019-09-24T10:14:05Z" level=info msg="Disregard the error. No existing Identity provider deployment found. Generating admin username" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with Keycloak admin username: admin" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with Postgres User: pgche" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with Postgres DB: dbche" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with Postgres hostname: postgres" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with Postgres port: 5432" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with installation flavor: che" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with Keycloak realm: che" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with Keycloak client ID: che-public" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with log level: INFO" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Updating eclipse-che CR with debug: false" 
time="2019-09-24T10:14:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object Service, name: postgres" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object PersistentVolumeClaim, name: postgres-data" 
time="2019-09-24T10:14:05Z" level=info msg="Waiting for PVC postgres-data to be bound. Default timeout: 10 seconds" 
time="2019-09-24T10:14:05Z" level=info msg="PVC postgres-data successfully bound to volume pvc-c3c80bea-316f-4be5-8081-dfe9aca1508e" 
time="2019-09-24T10:14:05Z" level=info msg="Creating a new object: Deployment, name: postgres" 
time="2019-09-24T10:14:06Z" level=info msg="Waiting for deployment postgres. Default timeout: 420 seconds" 
time="2019-09-24T10:14:55Z" level=info msg="Deployment 'postgres' successfully scaled to 1" 
time="2019-09-24T10:14:55Z" level=info msg="Running exec to create Keycloak DB, user, privileges in pod postgres-688bcffd85-btv94" 
time="2019-09-24T10:14:56Z" level=info msg="Exec successfully completed" 
time="2019-09-24T10:14:56Z" level=info msg="Updating eclipse-che CR with status: provisioned with DB and user: true" 
time="2019-09-24T10:14:56Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:56Z" level=info msg="Creating a new object Service, name: che-host" 
time="2019-09-24T10:14:56Z" level=info msg="Creating a new object Ingress, name: che" 
time="2019-09-24T10:14:56Z" level=info msg="Updating eclipse-che CR with CheHost URL: che-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:14:56Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:56Z" level=info msg="Creating a new object Service, name: keycloak" 
time="2019-09-24T10:14:56Z" level=info msg="Creating a new object Ingress, name: keycloak" 
time="2019-09-24T10:14:56Z" level=info msg="Updating eclipse-che CR with Keycloak URL: http://keycloak-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:14:56Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:14:56Z" level=info msg="Creating a new object: Deployment, name: keycloak" 
time="2019-09-24T10:14:57Z" level=info msg="Waiting for deployment keycloak. Default timeout: 420 seconds" 
time="2019-09-24T10:15:36Z" level=info msg="Deployment 'keycloak' successfully scaled to 1" 
time="2019-09-24T10:15:36Z" level=info msg="Running exec to create realm, client and user in pod keycloak-55b98789b-44nrz" 
time="2019-09-24T10:15:48Z" level=info msg="Exec successfully completed" 
time="2019-09-24T10:15:48Z" level=info msg="Updating eclipse-che CR with status: provisioned with Keycloak: true" 
time="2019-09-24T10:15:48Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:15:48Z" level=info msg="Creating a new object Ingress, name: devfile-registry" 
time="2019-09-24T10:15:48Z" level=info msg="Creating a new object Service, name: devfile-registry" 
time="2019-09-24T10:15:48Z" level=info msg="Creating a new object: Deployment, name: devfile-registry" 
time="2019-09-24T10:15:49Z" level=info msg="Waiting for deployment devfile-registry. Default timeout: 420 seconds" 
time="2019-09-24T10:16:50Z" level=info msg="Deployment 'devfile-registry' successfully scaled to 1" 
time="2019-09-24T10:16:50Z" level=info msg="Updating eclipse-che CR with status: Devfile Registry URL: http://devfile-registry-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:16:50Z" level=warning msg="Failed to update eclipse-che CR. Fetching the latest CR version: Operation cannot be fulfilled on checlusters.org.eclipse.che "eclipse-che": the object has been modified; please apply your changes to the latest version and try again" 
time="2019-09-24T10:16:54Z" level=info msg="Updating eclipse-che CR with status: Devfile Registry URL: http://devfile-registry-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:16:54Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:16:54Z" level=info msg="Creating a new object Ingress, name: plugin-registry" 
time="2019-09-24T10:16:54Z" level=info msg="Creating a new object Service, name: plugin-registry" 
time="2019-09-24T10:16:54Z" level=info msg="Creating a new object: Deployment, name: plugin-registry" 
time="2019-09-24T10:16:55Z" level=info msg="Waiting for deployment plugin-registry. Default timeout: 420 seconds" 
time="2019-09-24T10:17:10Z" level=info msg="Deployment 'plugin-registry' successfully scaled to 1" 
time="2019-09-24T10:17:10Z" level=info msg="Updating eclipse-che CR with status: Plugin Registry URL: http://plugin-registry-op-test.192.168.99.127.nip.io/v3" 
time="2019-09-24T10:17:10Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:17:10Z" level=info msg="Creating a new object: ConfigMap, name: che" 
time="2019-09-24T10:17:10Z" level=info msg="Creating a new object: ConfigMap, name: custom" 
time="2019-09-24T10:17:10Z" level=info msg="Creating a new object: Deployment, name: che" 
time="2019-09-24T10:17:11Z" level=info msg="Updating eclipse-che CR with status: Che API: Unavailable" 
time="2019-09-24T10:17:11Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:17:11Z" level=info msg="Waiting for deployment che. Default timeout: 420 seconds" 
time="2019-09-24T10:20:12Z" level=info msg="Deployment 'che' successfully scaled to 1" 
time="2019-09-24T10:20:12Z" level=info msg="Updating eclipse-che CR with Keycloak URL status: http://keycloak-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:20:12Z" level=warning msg="Failed to update eclipse-che CR. Fetching the latest CR version: Operation cannot be fulfilled on checlusters.org.eclipse.che "eclipse-che": the object has been modified; please apply your changes to the latest version and try again" 
time="2019-09-24T10:20:18Z" level=info msg="Updating eclipse-che CR with Keycloak URL status: http://keycloak-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:20:18Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:20:18Z" level=info msg="Updating eclipse-che CR with status: Eclipse Che server: Available" 
time="2019-09-24T10:20:18Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:20:18Z" level=info msg="Updating eclipse-che CR with Eclipse Che server URL: http://che-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:20:18Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:20:18Z" level=info msg="Eclipse Che is now available at: http://che-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:20:18Z" level=info msg="Updating eclipse-che CR with version: 7.1.0" 
time="2019-09-24T10:20:18Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with auto-generated CheCluster DB password: password-hidden" 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with Keycloak admin username: " 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with Postgres User: pgche" 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with Postgres DB: dbche" 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with Postgres hostname: postgres" 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with Postgres port: 5432" 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with installation flavor: che" 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with Keycloak realm: che" 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:01Z" level=info msg="Updating eclipse-che CR with Keycloak client ID: che-public" 
time="2019-09-24T10:22:01Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:02Z" level=info msg="Updating eclipse-che CR with Keycloak URL: http://keycloak-op-test.192.168.99.127.nip.io" 
time="2019-09-24T10:22:02Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:06Z" level=info msg="Updating che Deployment with image eclipse/che-server:nightly" 
time="2019-09-24T10:22:06Z" level=info msg="Updating eclipse-che CR with version: nightly" 
time="2019-09-24T10:22:06Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:06Z" level=info msg="Updating che ConfigMap" 
time="2019-09-24T10:22:07Z" level=info msg="Updating eclipse-che CR with Keycloak admin username: " 
time="2019-09-24T10:22:07Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:12Z" level=info msg="Deployment che is in the rolling update state" 
time="2019-09-24T10:22:12Z" level=info msg="Updating eclipse-che CR with status: Available: Rolling update in progress" 
time="2019-09-24T10:22:12Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:22:12Z" level=info msg="Waiting for a successful rolling update of deployment che. Default timeout: 420 seconds" 
time="2019-09-24T10:29:12Z" level=info msg="Updating eclipse-che CR with Keycloak admin username: " 
time="2019-09-24T10:29:12Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:29:17Z" level=info msg="Deployment che is in the rolling update state" 
time="2019-09-24T10:29:17Z" level=info msg="Updating eclipse-che CR with status: Available: Rolling update in progress" 
time="2019-09-24T10:29:17Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:29:17Z" level=info msg="Waiting for a successful rolling update of deployment che. Default timeout: 420 seconds" 
time="2019-09-24T10:36:17Z" level=info msg="Updating eclipse-che CR with Keycloak admin username: " 
time="2019-09-24T10:36:17Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:36:22Z" level=info msg="Deployment che is in the rolling update state" 
time="2019-09-24T10:36:22Z" level=info msg="Updating eclipse-che CR with status: Available: Rolling update in progress" 
time="2019-09-24T10:36:23Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-09-24T10:36:23Z" level=info msg="Waiting for a successful rolling update of deployment che. Default timeout: 420 seconds" 

The update of the CR that causes the failure starts at 2019-09-24T10:22:01Z

[davidfestal]

So, this bug doesn't happen anytime an existing Custom Resource is changed. It only happens if you reset the password fields of an existing, already-reconciled CheCluster Custom Resource, by running kubectly apply -f my-changed-cr.yaml for example, as @metlos shown in previous comment:

sed -i "s/cheImageTag: ''/cheImageTag: 'nightly'/" cr.yaml
kubectl apply -f cr.yaml -n=op-test

The main reason is that, when you apply the changed CR again from the yaml file you have locally, you override the Postgres and Keycloak passwords to empty strings. This triggers the generation of new passwords by the operator, and brings the bug, because the password change is not detected and the Postgres POD is not restarted.

For now the preferred way to change an existing custom resource is to patch it, instead of applying it.