Use the RocksDB WriteBatchWithIndex to implement the read-your-own-writes in transaction

[git-hulk]

• This closes Atomic commit write batches in the command EXEC #1281
• This closes [BUG] Transaction can't guarantee atomicity #487

By default, the writes cannot be seen in transactions if the WriteBatch
wasn't committed. To mitigate this issue, RocksDB offers the WriteBatchWithIndex
to make the uncommitted writes visible if needed.

For more information can see: https://rocksdb.org/blog/2015/02/27/write-batch-with-index.html

This PR looks not easy to review at first glance, but it will be easier if review commits one by one:

• The first commit adds the function GetWriteBatch in storage to wrap the WriteBatch for the write operations, making it possible to group and write at once in the EXEC command
• The second commit supports using WriteBatchIndex in transaction mode so that those uncommitted writes can be seen in the same transaction
• The third commit replaces the DB's Get and NewIterator API with WriteBatchIndex to allow visit its own writes in transaction mode.

[mapleFU]

I didn't review it carefully, so maybe I made a mistake, but my point is that this changes too much.

Assume that a transaction is execution, it doesn't block all execution threads, so, the things would be trickey:

1. Storage (which is shared among system) would be mark as txn. My point is that if multiple thread call exec, which would win?
2. When an non-txn get executing when txn is executing, what would it happens?

I suggest that rename it to ctx, and rather than save txn flag to storage, encapsulate it into a per-request context. And rather than transaction, this maybe better called a batch, because maybe no conflict detection here.

Updated:

Seems that exclusive flag would make exec holds and write lock. But still argue that would a exception cause the txn abort.

[mapleFU]

Other LGTM. Though I still think put txn_write_batch_ in storage in a bit trickey, can we add some description here, for it can only used by exclusive command?

[git-hulk]

Other LGTM. Though I still think put txn_write_batch_ in storage in a bit trickey, can we add some description here, for it can only used by exclusive command?

Thanks for your kind review. Agreed to add more comments to explain why the current implementation is right. I'm also thinking about how to allow multi transactions to improve performance because it's unnecessary to be exclusive command if we can promise the commit all or none. But can retrospect this after this PR and the watch/unwatch feature are ready.

[mapleFU]

Just parallel exec can be put into a "Session.Context" rather than Storage. A exec can write to it own session, and read first it own session.

Though it's ok, but it needs detail considering. For know, I think txn related is all under exclusive mode, because they only used by "Exec", which is exclusive.

[git-hulk]

Just parallel exec can be put into a "Session.Context" rather than Storage. A exec can write to it own session, and read first it own session.

Yes, the transaction state indeed is a connection state, but it needs some refactoring work to support binding transaction conetext in underlying storage type implementation. We won't do that until this PR and watch/unwatch are supported.

[torwig]

In general, LGTM.

[mapleFU]

Other LGTM here, but I still think we need more log and comments.

[mapleFU]

Seems that incoming changes is that write with txn_batch_mode_ changed to writeToDb, and don't rely on txn_batch_mode_.

Currently I've no problem, but I hope we can moke error response to test this in the future. Maybe you can use sync_point in rocksdb ( I've port one of it: https://github.com/mapleFU/sync_point , but you can use it in rocksdb) to mock partial exec error in testing.

[torwig]

LGTM

[caipengbo]

LGTM, a clear implementation.

[git-hulk]

Thanks all, merging…