localstore reserve logic #1322

zelig · 2021-02-24T02:16:24Z

the pr also fixes an index leak which happened (ModeGetRequest) when an as yet unsynced chunk was requested. It was put to GC incorrectly but moreover this entry was not removed when the chunk was synced (setSync was called)

#1346

AP https://hackmd.io/9ejVZEvSQg69zXG91YsFEQ

pkg/localstore/localstore.go

pkg/localstore/mode_put.go

pkg/localstore/postage.go

pkg/node/node.go

zelig · 2021-04-10T21:23:58Z

pkg/localstore/mode_put.go

+	}
+	item.Radius = item2.Radius
+
+	if withinRadiusFn(db, item) || forcePin {


when this is from a simple setsync call we also should not set the access timestamp to now.
only if it is putRequest.

but surely not ike this, instead:
make access update consistent irrespective of reserve/cache status. (as reserve can be evicted after a second and then will be deleted even before their earlier cached (and earlier accessed) sister chunks.

pkg/localstore/reserve.go

zelig · 2021-04-10T21:40:40Z

pkg/localstore/reserve.go

+		if err != nil {
+			return false, err
+		}
+		err = db.pullIndex.DeleteInBatch(batch, item2)


this is not consistent. The pullindex must include chunk batches otherwise nothing will be in the pullindex from any bin outside the neighbourhood radius

pkg/localstore/localstore.go

pkg/localstore/mode_get.go

zelig · 2021-04-12T20:59:32Z

pkg/localstore/mode_put.go

+		return false, 0, err
+	}
+
+	item.AccessTimestamp = now()


TODO: this is to be.

pkg/localstore/mode_put.go

zelig · 2021-04-12T21:10:01Z

pkg/localstore/mode_set.go

-	}
-	if !ok {
-		err = db.gcIndex.PutInBatch(batch, item)
+		item.AccessTimestamp = now()


TODO: here please take note of this. This should leave the access timestamp alone.

and the following line also dropped

....don't understand.... i thought you said setSync should update access timestamp

let's please have a writeup or at least a truth table about what happens to which index under which circumstances. i find documenting this in the code not the best thing to do and it just pollutes the code with hypothetical reminders. let's start with defining, then make changes to the codebase and in the meanwhile try to keep it clean

here it is quite simple, you update accesstimestamp if and only if the chunk iis requested

truth table?

pkg/localstore/reserve_test.go

zelig

lgtm

…is set after batch is forgotten

acud · 2021-04-13T11:30:41Z

famous last words?

postage: new pkg for postage stamps, uploader stamping (#890) * postage: new pkg for postage stamps, uploader stamping * postage: amount->value, blockNumber big.Int-> uint64, stamp only has batch ID, not Batch * postage: fix godoc and copyright * postage, swarm: - swarm.Stamp as an interface - add postage/testing for mock Stamps - fix Stamp MarshalBinary to allow nil batch id and signature - add StampSize const * postage: heed review feedback Co-authored-by: acud <[email protected]> Storage incentives: add stamper putter to api Postage BatchStore and BatchService (#1070) Co-authored-by: zelig <[email protected]> add normalisedBalance to updater interface (#1108) make value the normalised balance (#1111) postage: add event listener (#1099) Wire up postage stamp syncing (#1114) localstore, shed: persist stamps (#1116) add --postage to beeinfra.sh setup pullsync, pushsync: add postage stamps (#1117) postage: add create endpoint (#1142) retrieve erc20 address from postage contract (#1169) postage: check balance before attempting stamp creation (#1177) postage: fix bucket depth (#1178) api: use hex encoding in postage api (#1179) increase page size (#1182) postage: handle bucket depth error in api (#1183) localstore: attach stamp to outgoing chunk (#1192) update postage stamp contract addresses for new token (#1208) batchstore: reserve (#1262) * postage/batchstore: reserve logic Co-authored-by: acud <[email protected]> stamp support in storage and protocols (#1321) api: endpoints for stamp issuers (#1535) retrieval: add stamps (#1552) localstore reserve logic (#1322) Co-authored-by: acud <[email protected]>

zelig added the builds on open pr builds on another open one label Feb 24, 2021

zelig self-assigned this Feb 24, 2021

bee-runner bot added the pull-request label Feb 24, 2021

zelig mentioned this pull request Feb 24, 2021

localstore: preliminary changes related to postage stamps/batches #933

Closed

zelig force-pushed the postage-stamp-storage branch 2 times, most recently from 1694c83 to c020c7a Compare February 24, 2021 02:37

zelig mentioned this pull request Feb 24, 2021

localstore: simplify and fix #1063

Closed

zelig force-pushed the postage-stamp-storage branch from c020c7a to de15213 Compare February 24, 2021 05:40

zelig force-pushed the postage-localstore-reserve branch 4 times, most recently from 229d470 to a51dc04 Compare February 24, 2021 14:36

zelig force-pushed the postage-stamp-storage branch 2 times, most recently from f850603 to 435d47b Compare March 15, 2021 01:01

zelig force-pushed the postage-localstore-reserve branch from a51dc04 to 9b978c0 Compare March 15, 2021 21:08

acud force-pushed the postage-stamp-storage branch from 435d47b to e326684 Compare March 31, 2021 13:01

acud force-pushed the postage-localstore-reserve branch 2 times, most recently from db8ddaa to 8cecb25 Compare April 2, 2021 11:58

acud reviewed Apr 2, 2021

View reviewed changes

pkg/localstore/localstore.go Outdated Show resolved Hide resolved

acud reviewed Apr 2, 2021

View reviewed changes

pkg/localstore/mode_put.go Show resolved Hide resolved

Base automatically changed from postage-stamp-storage to storage-incentives April 2, 2021 12:11

acud force-pushed the postage-localstore-reserve branch from b98122c to bfc6fcc Compare April 2, 2021 12:18

acud removed the builds on open pr builds on another open one label Apr 2, 2021

zelig commented Apr 5, 2021

View reviewed changes

pkg/localstore/postage.go Outdated Show resolved Hide resolved

pkg/node/node.go Show resolved Hide resolved

acud force-pushed the postage-localstore-reserve branch 4 times, most recently from 83e4fdc to 32330ce Compare April 9, 2021 13:37

acud added ready for review The PR is ready to be reviewed and removed pull-request labels Apr 9, 2021

acud added 6 commits April 10, 2021 19:20

lint

35b5b17

fix tests

eb2aaa1

wip: change index structure to persist batch radius

db31c2a

reserve tests passing

4a182a8

fix localstore tests

f2c75b2

fix pushsync tests

7bc51be

acud force-pushed the postage-localstore-reserve branch from 4506b4d to 7bc51be Compare April 10, 2021 17:21

acud added 2 commits April 10, 2021 20:15

postage: revert changes no longer needed

379fc49

localstore: cleanup, fix ModePutRequestPin, add copy

5d15bac

zelig commented Apr 10, 2021

View reviewed changes

acud and others added 5 commits April 11, 2021 17:54

storage: add ModePutRequestCache

7799d1c

wip

032b28a

wip

01c2e92

fix tests, revert chunk prioritization under certain circumstances

1f2e2be

update gc index on get only if it already exists in index

291368e

zelig commented Apr 12, 2021

View reviewed changes

pr comments

23c1a7d

zelig commented Apr 13, 2021

View reviewed changes

acud added 3 commits April 13, 2021 12:02

add cleanup for postageRadiusIndex and fallback case for chunk which …

6b4f40d

…is set after batch is forgotten

fix tests

e5401c0

minor aesthetics

0e14362

acud approved these changes Apr 13, 2021

View reviewed changes

acud merged commit 2ee3530 into storage-incentives Apr 13, 2021

acud deleted the postage-localstore-reserve branch April 13, 2021 11:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

localstore reserve logic #1322

localstore reserve logic #1322

zelig commented Feb 24, 2021 •

edited

Loading

zelig Apr 10, 2021

zelig Apr 10, 2021

zelig Apr 12, 2021

acud Apr 13, 2021

zelig Apr 12, 2021

zelig Apr 12, 2021

acud Apr 13, 2021

acud Apr 13, 2021

zelig Apr 13, 2021

acud Apr 13, 2021

zelig left a comment

acud commented Apr 13, 2021

localstore reserve logic #1322

localstore reserve logic #1322

Conversation

zelig commented Feb 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zelig left a comment

Choose a reason for hiding this comment

acud commented Apr 13, 2021

zelig commented Feb 24, 2021 •

edited

Loading