You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This bug was encountered while using GEDS-HDFS as a tier-2 storage for Pravega.
While configured to spill to MInIO (S3), GEDS spills earlier than expected in the event of an MInIO outage. with the working directory set to a drive with 20GB of storage space the expected behaviour is that in the event of an MInIO outage, GEDS should fill up to ~70% of its capacity (~14GB), before throttling and errors are encountered.
In reality, only ~2.4GB is written to GEDS before throttling occurs.
In the logs, cURL error 7 (could not connect) and 28 (Timeout reached) are shown repeatedly. In particular, the first instance of error 28 aligns with when the throttling begins.
I believe GEDS may be able to last significantly longer while under an MInIO outage, and this is being hindered by some sort of timeout.
java.util.concurrent.CompletionException: io.pravega.segmentstore.storage.chunklayer.ChunkStorageException: checkExists
at io.pravega.segmentstore.storage.chunklayer.AsyncBaseChunkStorage.lambda$execute$13(AsyncBaseChunkStorage.java:751)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at io.pravega.common.concurrent.ThreadPoolScheduledExecutorService$ScheduledRunnable.run(ThreadPoolScheduledExecutorService.java:209)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.pravega.segmentstore.storage.chunklayer.ChunkStorageException: checkExists
at io.pravega.storage.hdfs.HDFSChunkStorage.convertException(HDFSChunkStorage.java:367)
at io.pravega.storage.hdfs.HDFSChunkStorage.checkExists(HDFSChunkStorage.java:169)
at io.pravega.segmentstore.storage.chunklayer.BaseChunkStorage.lambda$checkExistsAsync$3(BaseChunkStorage.java:89)
at io.pravega.segmentstore.storage.chunklayer.AsyncBaseChunkStorage.lambda$execute$13(AsyncBaseChunkStorage.java:747)
... 6 common frames omitted
Caused by: java.io.IOException: Unable to file status: _system/containers/_sysjournal.container4.snapshot_info: curlCode: 7, Couldn't connect to server
at com.ibm.geds.GEDS.nativeStatus(Native Method)
at com.ibm.geds.GEDS.status(GEDS.java:260)
at com.ibm.geds.hdfs.GEDSHadoopFileSystem.getFileStatus(GEDSHadoopFileSystem.java:154)
at io.pravega.storage.hdfs.HDFSChunkStorage.checkExists(HDFSChunkStorage.java:164)
... 8 common frames omitted
_system/containers/_sysjournal.container4.snapshot_info
Curl Code 28
at io.pravega.segmentstore.storage.chunklayer.AsyncBaseChunkStorage.lambda$execute$13(AsyncBaseChunkStorage.java:751)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at io.pravega.common.concurrent.ThreadPoolScheduledExecutorService$ScheduledRunnable.run(ThreadPoolScheduledExecutorService.java:209)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.pravega.segmentstore.storage.chunklayer.ChunkStorageException: checkExists
at io.pravega.storage.hdfs.HDFSChunkStorage.convertException(HDFSChunkStorage.java:367)
at io.pravega.storage.hdfs.HDFSChunkStorage.checkExists(HDFSChunkStorage.java:169)
at io.pravega.segmentstore.storage.chunklayer.BaseChunkStorage.lambda$checkExistsAsync$3(BaseChunkStorage.java:89)
at io.pravega.segmentstore.storage.chunklayer.AsyncBaseChunkStorage.lambda$execute$13(AsyncBaseChunkStorage.java:747)
... 6 common frames omitted
Caused by: java.io.IOException: Unable to file status: _system/containers/_sysjournal.container7.snapshot_info: curlCode: 28, Timeout was reached
at com.ibm.geds.GEDS.nativeStatus(Native Method)
at com.ibm.geds.GEDS.status(GEDS.java:260)
at com.ibm.geds.hdfs.GEDSHadoopFileSystem.getFileStatus(GEDSHadoopFileSystem.java:154)
at io.pravega.storage.hdfs.HDFSChunkStorage.checkExists(HDFSChunkStorage.java:164)
... 8 common frames omitted
_system/containers/_sysjournal.container7.snapshot_info```
The text was updated successfully, but these errors were encountered:
Describe the bug
To Reproduce
/setup-scripts/pravega-geds-install.sh
to install the GEDS-integrated Pravega deployment on Kubernetes./experiment
and runrun-experiment.sh
.kubectl logs pravega-pravega-segmentstore-0
. The error(s) should be visible in the logs.Additional information
Configuration Used:
GEDS is configured using environment variables:
Curl Code 7
Curl Code 28
The text was updated successfully, but these errors were encountered: