feat: Add document delete mechanics

[fredcarle]

Relevant issue(s)

Resolves #1262
Resolves #867

Description

This PR covers deleting (not purging) of documents. It adds a status value to the composite block and allows the merkle DAG to keep track of deleted documents. It also moves the fields from value instances to deleted instances to easily filter deleted documents and avoid extra overhead from the deleted documents on normal queries. We can thus select to show deleted documents when querying and are able to show the status of the document (_isDeleted: true/false) in the response.

Tasks

• I made sure the code is well commented, particularly hard-to-understand areas.
• I made sure the repository-held documentation is changed accordingly.
• I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
• I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

(replace) Describe the tests performed to verify the changes. Provide instructions to reproduce them.

Specify the platform(s) on which this was tested:

• (modify the list accordingly)
• Arch Linux
• Debian Linux
• MacOS
• Windows

[AndrewSisley]

way out of scope, but I just noticed this is not linked to an open ticket, but the one partially responsible for its introduction. We need to open a ticket for this.

[AndrewSisley]

#1264

[AndrewSisley]

Got a few items that I think should be discussed before merge.

I'm sorry as I feel like you checked in with me on this quite early, I misunderstood you then without realising and now it is needed to be talked about within the PR.

I haven't fully reviewed the internals/tests for this either, I'll go over them once we've all agreed on the high level discussion items.

[jsimnz]

Some issues/thoughts.

Haven't gone through everything yet, but I need to go to sleep :D

Ill ping when im up and we can keep progressing.

[codecov]

Codecov Report

Merging #1263 (7dd4293) into develop (b9d2e32) will decrease coverage by 0.05%.
The diff coverage is 69.52%.

@@             Coverage Diff             @@
##           develop    #1263      +/-   ##
===========================================
- Coverage    70.27%   70.22%   -0.05%     
===========================================
  Files          184      184              
  Lines        17700    17791      +91     
===========================================
+ Hits         12438    12494      +56     
- Misses        4324     4346      +22     
- Partials       938      951      +13     

Impacted Files	Coverage Δ
client/request/select.go	100.00% <ø> (ø)
core/doc.go	96.72% <ø> (ø)
db/errors.go	81.17% <ø> (ø)
db/fetcher/encoded_doc.go	61.87% <ø> (-0.28%)	⬇️
planner/mapper/targetable.go	72.97% <0.00%> (-2.03%)	⬇️
planner/planner.go	77.85% <ø> (ø)
core/crdt/composite.go	61.53% <38.63%> (-14.47%)	⬇️
db/collection.go	68.26% <60.78%> (-0.59%)	⬇️
core/crdt/lwwreg.go	75.25% <70.00%> (-1.41%)	⬇️
db/fetcher/fetcher.go	62.88% <70.27%> (+1.92%)	⬆️
... and 13 more

... and 4 files with indirect coverage changes

[jsimnz]

Really great stuff! Super excited to see this land.

Small number of todos/suggestions, few more questions/thoughts.

Going to approve now, so you dont need to play another round of reviewer ping pong, but do go through the various issues 👍

requested changes covered