refactor: Strongly typed key refactor (#17) #84

AndrewSisley · 2021-12-08T21:28:13Z

Closes #17 Closes #18

Updated review of the original PR (below), against develop branch
#19

Suggest re-reviewing properly once we know how this affects performance, it got quite stale and it is possible that I have added in some strangeness in places with the time-off and/or rebase onto latest develop

Also updates the composite dag cids to use the dag-pb codec to make the consistent with the field cids

Benchmarks have been run, suggest little-to-no change.

todo · 2021-12-08T21:28:16Z

paramaterize ns/suffix

defradb/core/crdt/base.go

Lines 30 to 35 in 687c223

    
           // @TODO paramaterize ns/suffix 
        
           func newBaseCRDT(store core.DSReaderWriter, key core.DataStoreKey) baseCRDT { 
        
           	return baseCRDT{ 
        
           		store: store, 
        
           		key:   key, 
        
           	}

This comment was generated by todo based on a `TODO` comment in `687c223` in #84. cc @sourcenetwork.

todo · 2021-12-08T21:28:19Z

Support multiple spans

defradb/db/fetcher/fetcher.go

Lines 114 to 117 in 687c223

    
           	Prefix: spans[0].Start().ToString(), // @todo: Support multiple spans 
        
           } 
        
           if df.reverse { 
        
           	q.Orders = []dsq.Order{dsq.OrderByKeyDescending{}}

This comment was generated by todo based on a `todo` comment in `687c223` in #84. cc @sourcenetwork.

todo · 2021-12-08T21:28:22Z

Config logger param package wide

defradb/merkle/clock/clock.go

Lines 42 to 45 in 687c223

    
           		headset:   newHeadset(headstore, namespace), //TODO: Config logger param package wide 
        
           		crdt:      crdt, 
        
           	} 
        
           }

This comment was generated by todo based on a `TODO` comment in `687c223` in #84. cc @sourcenetwork.

codecov · 2022-01-26T14:33:43Z

Codecov Report

Merging #84 (7d635b4) into develop (8dd342f) will increase coverage by 0.60%.
The diff coverage is 87.46%.

@@             Coverage Diff             @@
##           develop      #84      +/-   ##
===========================================
+ Coverage    57.47%   58.07%   +0.60%     
===========================================
  Files          103      103              
  Lines        10045    10206     +161     
===========================================
+ Hits          5773     5927     +154     
- Misses        3641     3645       +4     
- Partials       631      634       +3

Impacted Files	Coverage Δ
merkle/crdt/merklecrdt.go	`26.08% <ø> (ø)`
query/graphql/planner/versionedscan.go	`0.00% <0.00%> (ø)`
document/field.go	`61.11% <50.00%> (ø)`
query/graphql/planner/select.go	`73.47% <50.00%> (ø)`
db/fetcher/versioned.go	`49.18% <62.50%> (+0.27%)`	⬆️
db/collection_update.go	`42.10% <66.66%> (ø)`
db/fetcher/dag.go	`57.31% <66.66%> (+0.49%)`	⬆️
db/sequence.go	`65.11% <66.66%> (ø)`
query/graphql/planner/dagscan.go	`72.45% <66.66%> (ø)`
store/wrappedStore.go	`61.36% <66.66%> (+0.07%)`	⬆️
... and 28 more

AndrewSisley · 2022-02-01T21:08:39Z

Locally run benchmarks against develop:


```goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/collection
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Collection_UserSimple_CreateMany_Sync_0_10-8             9     239646252 ns/op
Benchmark_Collection_UserSimple_CreateMany_Sync_0_100-8            1    2713297484 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_1-8                 66      20276092 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_10-8                10     225022955 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_100-8                1    2217967837 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_1000-8               1    29229935496 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_1-8                66      20609855 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_100-8               2    1326814788 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_1000-8              1    16803631883 ns/op

...

Benchmark_Planner_UserSimple_ParseQuery-8           2466        409320 ns/op
Benchmark_Planner_UserSimple_MakePlan-8             5420        192085 ns/op
PASS
ok      github.com/sourcenetwork/defradb/bench/query/planner    3.343s
goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/query/simple
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Query_UserSimple_Query_Sync_1-8                                555       2071934 ns/op
Benchmark_Query_UserSimple_Query_Sync_10-8                               121       9383499 ns/op
Benchmark_Query_UserSimple_Query_Sync_100-8                               15      72842972 ns/op
Benchmark_Query_UserSimple_Query_Sync_1000-8                               2     871541347 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_1-8                     339       3006475 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_10-8                     87      13092536 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_100-8                    13      88203815 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_1000-8                    2     917211043 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_1-8                337       3003535 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_10-8                85      13379456 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_100-8               50      21518782 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_1000-8              43      26146472 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_10-8                14      81920500 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_100-8               12      90098140 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_1000-8              10     107609424 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_1-8               330       3104380 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_10-8              135       9382646 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_100-8             103       9777702 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_1000-8             86      12409382 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_1-8                       315       3229584 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_10-8                       81      14130994 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_100-8                      12      97549490 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_1000-8                      1    1096847379 ns/op
PASS
ok      github.com/sourcenetwork/defradb/bench/query/simple 443.801s
goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/storage
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0064-8                94017         12190 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0128-8                96955         12718 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0256-8                90009         13115 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0512-8                85572         13771 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:1024-8                72784         16071 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0064-8                8890        121504 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0128-8                8868        125167 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0256-8                7738        131765 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0512-8                7972        134999 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:1024-8                6601        163579 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0064-8                961       1164513 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0128-8                921       1232905 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0256-8                915       1280763 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0512-8                867       1343811 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:1024-8                744       1593338 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0064-8              80648         13615 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0128-8              82878         14526 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0256-8              79682         15135 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0512-8              74599         16309 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:1024-8              65233         18405 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0064-8              7707        140614 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0128-8              7513        146161 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0256-8              7485        152235 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0512-8              6416        156983 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:1024-8              5950        179254 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0064-8              831       1446060 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0128-8              843       1495171 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0256-8              754       1472171 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0512-8              741       1622043 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:1024-8              663       1856769 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0064-8            6513        177718 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0128-8            6414        179410 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0256-8            6159        188714 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0512-8            5709        202788 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:1024-8            4972        233545 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0064-8           2718        444720 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0128-8           2509        476643 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0256-8           2188        545880 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0512-8           1800        665289 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:1024-8           1292        935881 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0064-8           411       2862848 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0128-8           374       3200378 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0256-8           310       3728676 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0512-8           235       4946712 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:1024-8           142       7548855 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0064-8          6398        177205 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0128-8          6342        181607 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0256-8          6138        187895 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0512-8          5652        203607 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:1024-8          4902        236237 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0064-8         2697        449182 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0128-8         2496        508059 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0256-8         2200        544070 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0512-8         1797        681540 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:1024-8         1276        971957 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0064-8                 380       3206424 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0128-8                 356       3692331 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0256-8                 312       3725360 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0512-8                 229       4922088 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:1024-8                 150       7758769 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0064-8                        9090        125182 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0128-8                        8908        128488 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0256-8                        8446        136146 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0512-8                        7729        150792 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:1024-8                        6522        177592 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0064-8                        951       1250454 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0128-8                        936       1288475 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0256-8                        882       1374813 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0512-8                        802       1502091 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:1024-8                        680       1765798 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0064-8                        93      12463592 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0128-8                        91      12953270 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0256-8                        86      13495263 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0512-8                        78      14958467 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:1024-8                        66      17697077 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0064-8                      9069        124922 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0128-8                      8817        128710 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0256-8                      8448        134179 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0512-8                      7563        151317 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:1024-8                      6318        178023 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0064-8                      962       1256876 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0128-8                      931       1288262 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0256-8                      884       1363950 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0512-8                      795       1496059 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:1024-8                      674       1775000 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0064-8                      92      12548428 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0128-8                      90      12941666 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0256-8                      85      13646710 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0512-8                      78      14931087 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:1024-8                      64      17853992 ns/op

AndrewSisley · 2022-02-01T21:09:16Z

Locally run benchmarks against this branch:

goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/collection
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Collection_UserSimple_CreateMany_Sync_0_10-8            10     236724399 ns/op
Benchmark_Collection_UserSimple_CreateMany_Sync_0_100-8            1    2665020029 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_1-8                 67      20447414 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_10-8                10     229109034 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_100-8                1    2238841258 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_1000-8               1    28987113465 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_1-8                66      21192607 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_100-8               2     968039845 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_1000-8              1    16129026256 ns/op

...

Benchmark_Planner_UserSimple_ParseQuery-8           2512        412315 ns/op
Benchmark_Planner_UserSimple_MakePlan-8             5808        202858 ns/op
PASS
ok      github.com/sourcenetwork/defradb/bench/query/planner    3.503s
goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/query/simple
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Query_UserSimple_Query_Sync_1-8                                554       2037645 ns/op
Benchmark_Query_UserSimple_Query_Sync_10-8                               123       9168520 ns/op
Benchmark_Query_UserSimple_Query_Sync_100-8                               15      76016816 ns/op
Benchmark_Query_UserSimple_Query_Sync_1000-8                               2     874972950 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_1-8                     339       2982455 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_10-8                     86      12840973 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_100-8                    13      88448445 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_1000-8                    2     895083153 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_1-8                403       2968050 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_10-8                87      12995957 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_100-8               50      21184619 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_1000-8              43      25910912 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_10-8                14      81657222 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_100-8               13      85815675 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_1000-8              10     107366396 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_1-8               330       3052546 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_10-8              147       9119578 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_100-8             109       9631562 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_1000-8             84      12450439 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_1-8                       312       3210856 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_10-8                       81      13878314 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_100-8                      12      98169760 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_1000-8                      1    1071953447 ns/op
PASS
ok      github.com/sourcenetwork/defradb/bench/query/simple 426.996s
goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/storage
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0064-8                91335         12591 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0128-8                90103         12167 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0256-8                89707         12752 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0512-8                85858         14102 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:1024-8                74266         16657 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0064-8                8856        123825 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0128-8                8700        122168 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0256-8                8236        130980 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0512-8                7858        138529 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:1024-8                6570        160983 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0064-8                967       1190543 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0128-8                992       1228939 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0256-8                904       1347925 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0512-8                840       1386468 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:1024-8                722       1663416 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0064-8              83026         14015 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0128-8              83110         13798 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0256-8              79153         14804 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0512-8              74569         15778 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:1024-8              57446         18495 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0064-8              7694        149704 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0128-8              6896        147433 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0256-8              7080        152505 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0512-8              6956        160450 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:1024-8              5881        185295 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0064-8              826       1498373 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0128-8              838       1524953 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0256-8              784       1498561 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0512-8              576       1800240 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:1024-8              639       1796060 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0064-8            6550        180627 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0128-8            6324        179240 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0256-8            6199        187416 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0512-8            5703        204171 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:1024-8            4977        234379 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0064-8           2738        449882 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0128-8           2526        486032 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0256-8           2222        544662 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0512-8           1788        660338 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:1024-8           1296       1017617 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0064-8           397       3232241 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0128-8           372       3164444 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0256-8           313       3754809 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0512-8           234       5407312 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:1024-8           153       7568293 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0064-8          6516        177887 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0128-8          6420        180718 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0256-8          6133        187860 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0512-8          5620        204556 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:1024-8          4861        235815 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0064-8         2686        449266 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0128-8         2529        479754 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0256-8         2200        544111 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0512-8         1800        659725 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:1024-8         1264        960688 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0064-8                 408       2846077 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0128-8                 375       3565962 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0256-8                 308       3759211 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0512-8                 234       5178633 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:1024-8                 150       7464825 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0064-8                        9162        124628 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0128-8                        8985        127645 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0256-8                        8469        134978 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0512-8                        7742        148031 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:1024-8                        6517        178467 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0064-8                        960       1237192 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0128-8                        860       1282484 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0256-8                        885       1371603 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0512-8                        802       1488171 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:1024-8                        666       1773916 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0064-8                        94      12500481 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0128-8                        91      12832877 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0256-8                        86      13641413 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0512-8                        76      15044176 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:1024-8                        64      17692089 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0064-8                      9110        124786 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0128-8                      8883        128607 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0256-8                      8461        136180 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0512-8                      7530        148426 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:1024-8                      6369        177349 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0064-8                      963       1254496 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0128-8                      932       1280217 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0256-8                      884       1348944 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0512-8                      799       1501421 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:1024-8                      668       1782321 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0064-8                      92      12504911 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0128-8                      90      12758601 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0256-8                      85      13499809 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0512-8                      78      14978242 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:1024-8                      66      17809809 ns/op

jsimnz

Quite the change, I know I reviewed it back when you first opened this, but still takes a minute to fully wrap my head around the hierarchy/structure.

In general it all looks good, except the relation between key.DocKey and core.DataStoreKey, made some comments about it on the code. Basically we've created weird dependancy resulting in dockey.Key.DocKey access patterns. I think key.DocKey should be relatively self contained.

ds.Key was embedded within a key.DocKey before just so we could easily use all the ds.Key methods, but since we are using a structured approach, we dont really need the general purpose methods of the ds.Key.

You should just see the ds.Key in the key.DocKey as a string type with some helper functions.

core/key.go

db/collection_get.go

db/db_test.go

db/tests/query/all_commits/simple_test.go

db/tests/query/latest_commits/simple_test.go

document/key/dockey.go

WithPriorityFlag is called within reg.getPriority and should not be the responsibility of the caller

Is not needed and complicates the code

WithValueFlag is called in writeObjectMarker

Is wasteful taking the whole struct (and initing one in most cases)

I'm not 100% sure what this button does, but I ran the benches again and numbers are comparable with current dev

AndrewSisley added area/query Related to the query component area/datastore Related to the datastore / storage engine system refactor This issue specific to or requires *notable* refactoring of existing codebases and components labels Dec 8, 2021

AndrewSisley requested review from jsimnz and shahzadlone December 8, 2021 21:28

AndrewSisley self-assigned this Dec 8, 2021

AndrewSisley linked an issue Dec 8, 2021 that may be closed by this pull request

Convert all instances of ds.Key to core.Key in both internal and external APIs #17

Closed

1 task

jsimnz added the blocked label Jan 19, 2022

AndrewSisley force-pushed the sisley/refactor/I17-key-refactor branch from 687c223 to fe8158c Compare January 26, 2022 14:31

AndrewSisley force-pushed the sisley/refactor/I17-key-refactor branch 4 times, most recently from abeb2d4 to 9c029cc Compare February 1, 2022 19:20

jsimnz previously requested changes Feb 2, 2022

View reviewed changes

AndrewSisley force-pushed the sisley/refactor/I17-key-refactor branch from 9c029cc to d6a3780 Compare February 2, 2022 02:38

AndrewSisley requested a review from jsimnz February 2, 2022 02:55

jsimnz added this to the DefraDB v0.2 milestone Feb 2, 2022

AndrewSisley force-pushed the sisley/refactor/I17-key-refactor branch 3 times, most recently from 5de70cb to e31998a Compare February 2, 2022 19:29

AndrewSisley mentioned this pull request Feb 2, 2022

Hefty disjoint between txn.go.newTxn/Txn and db.go.NewDB/DB #172

Closed

1 task

jsimnz removed the blocked label Feb 2, 2022

jsimnz modified the milestones: DefraDB v0.2, DefraDB v0.3 Feb 2, 2022

AndrewSisley mentioned this pull request Feb 9, 2022

Clarify core vs client package differences #200

Closed

AndrewSisley force-pushed the sisley/refactor/I17-key-refactor branch 4 times, most recently from c2351e2 to 821203f Compare February 15, 2022 21:43

AndrewSisley force-pushed the sisley/refactor/I17-key-refactor branch from 821203f to 5dc940d Compare February 23, 2022 19:27

AndrewSisley added 11 commits March 3, 2022 14:53

Reference correct Key type in documentation

1e5ef18

Remove commented out import

405f5bb

Remove commented out import

c689109

Remove commented out code

8d6701e

Remove commented out import

fe89a15

Replace references to ds.Key with core.Key

9a5e30e

Replace ds.Key with structured data type

d464897

Remove duplicate call

1fa4c1b

WithPriorityFlag is called within reg.getPriority and should not be the responsibility of the caller

Remove DataStoreKey from Collection struct

0857a12

Is not needed and complicates the code

Remove duplicate call

73d5cc3

WithValueFlag is called in writeObjectMarker

Take docKey string as param instead of DataStoreKey

9744349

Is wasteful taking the whole struct (and initing one in most cases)

AndrewSisley force-pushed the sisley/refactor/I17-key-refactor branch from 5dc940d to 9744349 Compare March 3, 2022 20:08

Documet commit key changes

7d635b4

AndrewSisley requested a review from orpheuslummis March 4, 2022 14:06

shahzadlone approved these changes Mar 4, 2022

View reviewed changes

AndrewSisley merged commit 6bfcf7e into develop Mar 4, 2022

AndrewSisley deleted the sisley/refactor/I17-key-refactor branch March 4, 2022 19:30

AndrewSisley mentioned this pull request Jun 3, 2022

feat: Fetcher field selection and optimistic filter evaluation #491

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Strongly typed key refactor (#17) #84

refactor: Strongly typed key refactor (#17) #84

AndrewSisley commented Dec 8, 2021 •

edited

Loading

todo bot commented Dec 8, 2021

todo bot commented Dec 8, 2021

todo bot commented Dec 8, 2021

codecov bot commented Jan 26, 2022 •

edited

Loading

AndrewSisley commented Feb 1, 2022

AndrewSisley commented Feb 1, 2022

jsimnz left a comment

refactor: Strongly typed key refactor (#17) #84

refactor: Strongly typed key refactor (#17) #84

Conversation

AndrewSisley commented Dec 8, 2021 • edited Loading

todo bot commented Dec 8, 2021

paramaterize ns/suffix

This comment was generated by todo based on a TODO comment in 687c223 in #84. cc @sourcenetwork.

todo bot commented Dec 8, 2021

Support multiple spans

This comment was generated by todo based on a todo comment in 687c223 in #84. cc @sourcenetwork.

todo bot commented Dec 8, 2021

Config logger param package wide

This comment was generated by todo based on a TODO comment in 687c223 in #84. cc @sourcenetwork.

codecov bot commented Jan 26, 2022 • edited Loading

Codecov Report

AndrewSisley commented Feb 1, 2022

AndrewSisley commented Feb 1, 2022

jsimnz left a comment

Choose a reason for hiding this comment

AndrewSisley commented Dec 8, 2021 •

edited

Loading

This comment was generated by todo based on a `TODO` comment in `687c223` in #84. cc @sourcenetwork.

This comment was generated by todo based on a `todo` comment in `687c223` in #84. cc @sourcenetwork.

This comment was generated by todo based on a `TODO` comment in `687c223` in #84. cc @sourcenetwork.

codecov bot commented Jan 26, 2022 •

edited

Loading