Skip to content
This repository has been archived by the owner on Mar 3, 2023. It is now read-only.

Helm deployment break - WIP #3719

Open
windhamwong opened this issue Oct 18, 2021 · 2 comments
Open

Helm deployment break - WIP #3719

windhamwong opened this issue Oct 18, 2021 · 2 comments
Assignees
@windhamwong
Labels
helm Kubernetes zookeeper

Comments

@windhamwong
Copy link
Contributor

I create this issue for tracking the issue of, deploying Heron to Kubernetes with Helm, having issue with zookeeper.
Please feel free to leave your related case and log here.

TODO:
I am trying to solve the issue if this is a known issue.

The log of zookeeper is listed as below:

Validating enviornment
ZK_REPLICAS=3
MY_ID=1
ZK_LOG_LEVEL=INFO
ZK_DATA_DIR=/var/lib/zookeeper/data
ZK_DATA_LOG_DIR=/var/lib/zookeeper/log
ZK_LOG_DIR=var/log/zookeeper
ZK_CLIENT_PORT=2181
ZK_SERVER_PORT=2888
ZK_ELECTION_PORT=3888
ZK_TICK_TIME=2000
ZK_INIT_LIMIT=10
ZK_SYNC_LIMIT=5
ZK_MAX_CLIENT_CNXNS=60
ZK_MIN_SESSION_TIMEOUT=4000
ZK_MAX_SESSION_TIMEOUT=40000
ZK_HEAP_SIZE=256M
ZK_SNAP_RETAIN_COUNT=3
ZK_PURGE_INTERVAL=1
ENSEMBLE
server.1=heron-zookeeper-0.heron-zookeeper-headless.default.svc.cluster.local:2888:3888
server.2=heron-zookeeper-1.heron-zookeeper-headless.default.svc.cluster.local:2888:3888
server.3=heron-zookeeper-2.heron-zookeeper-headless.default.svc.cluster.local:2888:3888
Enviorment validation successful
rm: cannot remove '/opt/zookeeper/conf/zookeeper.conf': Permission denied
Creating ZooKeeper configuration
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 92: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 93: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 94: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 95: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 96: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 97: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 98: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 99: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 100: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 101: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 102: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 103: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 104: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 105: /opt/zookeeper/conf/zookeeper.conf: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 108: /opt/zookeeper/conf/zookeeper.conf: Permission denied
Wrote ZooKeeper configuration file to /opt/zookeeper/conf/zookeeper.conf
rm: cannot remove '/opt/zookeeper/conf/log4j.properties': Permission denied
Creating ZooKeeper log4j configuration
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 138: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 139: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 140: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 141: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 142: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 143: /opt/zookeeper/conf/log4j.properties: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 144: /opt/zookeeper/conf/log4j.properties: Permission denied
Wrote log4j configuration to /opt/zookeeper/conf/log4j.properties
Creating ZooKeeper data directories and setting permissions
mkdir: cannot create directory ‘var’: Permission denied
chown: invalid user: ‘zookeeper:zookeeper’
Created ZooKeeper data directories and set permissions in /var/lib/zookeeper/data
Creating JVM configuration file
Wrote JVM configuration to /opt/zookeeper/conf/java.env
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 151: /opt/zookeeper/conf/java.env: Permission denied
/opt/zookeeper/scripts/generate-zookeeper-config.sh: line 152: /opt/zookeeper/conf/java.env: Permission denied
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/conf/zookeeper.conf
mkdir: cannot create directory ‘data’: Permission denied
mkdir: cannot create directory ‘/opt/zookeeper/bin/../logs’: Permission denied
2021-10-18 10:22:57,409 [myid:] - INFO  [main:QuorumPeerConfig@135] - Reading configuration from: /opt/zookeeper/conf/zookeeper.conf
2021-10-18 10:22:57,415 [myid:] - WARN  [main:VerifyingFileFactory@59] - data/zookeeper is relative. Prepend ./ to indicate that you're sure!
2021-10-18 10:22:57,425 [myid:] - INFO  [main:QuorumPeerConfig@387] - clientPortAddress is 0.0.0.0:2181
2021-10-18 10:22:57,426 [myid:] - INFO  [main:QuorumPeerConfig@391] - secureClientPort is not set
2021-10-18 10:22:57,432 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2021-10-18 10:22:57,433 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1
2021-10-18 10:22:57,434 [myid:] - WARN  [main:QuorumPeerMain@125] - Either no config or no quorum defined in config, running  in standalone mode
2021-10-18 10:22:57,439 [myid:] - INFO  [main:ManagedUtil@45] - Log4j 1.2 jmx support found and enabled.
2021-10-18 10:22:57,439 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2021-10-18 10:22:57,455 [myid:] - INFO  [main:QuorumPeerConfig@135] - Reading configuration from: /opt/zookeeper/conf/zookeeper.conf
2021-10-18 10:22:57,459 [myid:] - WARN  [main:VerifyingFileFactory@59] - data/zookeeper is relative. Prepend ./ to indicate that you're sure!
2021-10-18 10:22:57,464 [myid:] - INFO  [PurgeTask:FileTxnSnapLog@115] - zookeeper.snapshot.trust.empty : false
2021-10-18 10:22:57,494 [myid:] - INFO  [main:QuorumPeerConfig@387] - clientPortAddress is 0.0.0.0:2181
2021-10-18 10:22:57,494 [myid:] - INFO  [main:QuorumPeerConfig@391] - secureClientPort is not set
2021-10-18 10:22:57,495 [myid:] - INFO  [main:ZooKeeperServerMain@117] - Starting server
2021-10-18 10:22:57,495 [myid:] - ERROR [PurgeTask:DatadirCleanupManager$PurgeTask@142] - Error occurred while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:127)
        at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:79)
        at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
        at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
        at java.base/java.util.TimerThread.run(Timer.java:506)
2021-10-18 10:22:57,499 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2021-10-18 10:22:57,495 [myid:] - INFO  [main:FileTxnSnapLog@115] - zookeeper.snapshot.trust.empty : false
2021-10-18 10:22:57,499 [myid:] - ERROR [main:ZooKeeperServerMain@75] - Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:127)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:124)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Unable to access datadir, exiting abnormally
@windhamwong windhamwong self-assigned this Oct 18, 2021
@nicknezis
Copy link
Contributor

nicknezis commented Oct 18, 2021 via email

@windhamwong
Copy link
Contributor Author

windhamwong commented Oct 22, 2021 •
edited
Loading

Confirmed that this happens when using helm, setting .Values.zookeeper.security := true.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@windhamwong windhamwong

Labels
helm Kubernetes zookeeper
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nicknezis @windhamwong