[BAHIR-183] [WIP] HDFS based MQTT client persistence #84
Conversation
lukasz-antoniak
force-pushed
the
BAHIR-183-2
branch
from
b5bf987
to
9709a62
Compare
|
Sorry to keep you wait, I will review this PR in a few days. |
|
yes |
|
@lukasz-antoniak what's the status of this pr, is this ready for final review/merge? |
|
@lresende, I would answer positive. This PR changes the way we implement HDFS persistence due to issues pointed out in JIRA commant. |
Very sorry to keep you wait. I will review it in weekends. |
|
|
||
| override def put(key: String, persistable: MqttPersistable): Unit = { | ||
| try { | ||
| val path = getPath(key) |
There was a problem hiding this comment.
I just worry about the performance of creating a file for each coming message.
|
And, I think we should better keep the ability to do flow-control, eg, control max number of message in each batch. |
As discussed in the JIRA ticket (https://issues.apache.org/jira/browse/BAHIR-183), revisited version of HDFS persistence of MQTT messages, which should ease recovery in case of device failure. The design discussion may be continued in JIRA, but I decided to open WIP PR for @yanlin-Lynn to review and comment.