Set more aggressive cache expiry values and run a Long-term storage test

[danieldaquino]

Damus: 1.6 (26) cc385d3

The current cache expiry values do not seem to be enough to keep storage usage low. I am opening this ticket to:

• Set more aggressive cache expiry values
• Manually monitor storage usage over time on at least one phone using Damus in real life.

[danieldaquino]

Sent a patch with more aggressive cache expiry values here: https://groups.google.com/a/damus.io/g/patches/c/rw_CHFh7nWo

I will start long term testing.

[danieldaquino]

Long-term test starts

Device: "Daniel's iPhone"

• Model: iPhone 13 mini
• iOS: 17 (Not putting the exact value as automatic iOS updates will likely happen during the test)
• Damus: Logged into the spreadsheet. Starting with 7744d3e8 (The version on the patch sent)

Procedure:

1. I started a spreadsheet to log test data (available here: https://docs.google.com/spreadsheets/d/1LjvKQReMXDKN_2PJ3kaXEhU8o--T9UhSrGpoKeqhKrI/edit?usp=sharing)
2. I setup a reminder for myself to log storage usage data every few days. (About 2~3 times per week)
3. I will keep going until we can draw some conclusions from the data
4. If anyone wants to participate in this test, please let me know and I can add write access to the spreadsheet.

@jb55, @alltheseas, I started this right away to give us more time to run this long term test, which might take a few days or weeks to get meaningful data patterns. Please let me know if you have questions or suggestions!

[alltheseas]

Sent a patch with more aggressive cache expiry values here: https://groups.google.com/a/damus.io/g/patches/c/rw_CHFh7nWo

I will start long term testing.

What's the expiry value? Is it duration based (e.g. 1 month)? Size based (e.g. 20 GB)?

[danieldaquino]

What's the expiry value? Is it duration based (e.g. 1 month)? Size based (e.g. 20 GB)?

@alltheseas it is currently duration-based, and depends on the type of image.

The current Damus Testflight version (1.6 (26) cc385d3) uses these values:

• Profile pictures: Never expire
• Banner pictures: Expire after 14 days
• Note pictures: Expire after 7 days

My patch is changing these to:

• Profile pictures: Expire after 60 days
• Banner pictures: Expire after 5 days
• Note pictures: Expire after 3 days

[alltheseas]

Tagging related tickets for future context.

user preference on storage #1554
users with less than 1 GB in storage available case: #1619

[alltheseas]

https://damus.io/note1uqg5wtaclzhefv3fhuw8xvmn2wu8guv5jlpxa8k2xvk0yks8x2uqkcyg9l

[danieldaquino]

@alltheseas, although the patch was merged, can I keep this one open for a bit longer? I am currently monitoring the storage usage over several days to study how it changes over time with the new expiry settings (https://docs.google.com/spreadsheets/d/1LjvKQReMXDKN_2PJ3kaXEhU8o--T9UhSrGpoKeqhKrI/edit?usp=sharing)

[alltheseas]

💯

[danieldaquino]

@jb55, @alltheseas, good news, the new cache expiry values seem to be bringing my storage usage down significantly!

I imagine this value will keep going down up until a certain point and then stabilize. I am curious as to what that value will be. I think it depends on how active the user is on the app, so it might vary depending on the user.

(Full data here)

I will keep this running to see where it goes

[alltheseas]

Good news.

For folks with less disk space I like the idea of a mechanism where you can set max storage. See example #1554

[jb55]

On Wed, Nov 15, 2023 at 10:06:11AM -0800, Daniel D’Aquino wrote: @jb55, @alltheseas, good news, the new cache expiry values seem to be bringing my storage usage down significantly! I imagine this value will keep going down up until a certain point and then stabilize. I am curious as to what that value will be. I think it depends on how active the user is on the app, so it might vary depending on the user.  (Full data [here](https://docs.google.com/spreadsheets/d/1LjvKQReMXDKN_2PJ3kaXEhU8o--T9UhSrGpoKeqhKrI/edit?usp=sharing))

awesome analysis. thank you!

[alltheseas]

Special case for future consideration: for folks with under 1 GB free space perpetual crash occurs #1619

[danieldaquino]

Long-term test quick update: It looks like I might have reached a storage usage plateau at around 5GB – 6GB.

(Full data here)

I will keep monitoring

[alltheseas]

I would set a 1GB max size on my limited phone.

Thanks!

[danieldaquino]

I would set a 1GB max size on my limited phone.

Thanks!

Yeah, it will be awesome if we can control the max storage size

[danieldaquino]

@alltheseas, @jb55, I believe my phone has settled at around 4GB — 6GB of storage use.

This experiment will probably start bringing diminishing returns in terms of valuable insights/data. Here are a couple of possible follow-up actions we can take from here:

• Set the cache values to be even more aggressive (if 4GB — 6GB is still a lot more than ideal)
• Get storage data from one of you (or someone else), to experiment if the settled storage usage value differs too much depending on the user
• Close this and we follow up later with Explore storage visualization, preferences, and cleaning #1554 as the next step in storage management, if we are more or less happy with how cache management is working right now.

Any preferences or thoughts on what to do next with this ticket?

[alltheseas]

Set the cache values to be even more aggressive (if 4GB — 6GB is still a lot more than ideal)

Do you think performance will be affected, and if so how, if the value is set to a more aggressive 1 GB?

[alltheseas]

@danieldaquino overall I think this is a successful test & implementation. Yegor had high 10s GB of cache, and was on track towards 100 GB if he was not low on disk space if I remember correctly. We can fine tune later, and close for now.

Well done 💪

[danieldaquino]

Well done 💪

Thank you!

Do you think performance will be affected, and if so how, if the value is set to a more aggressive 1 GB?

The problem is that we do not have a mechanism for setting a max storage size yet, unfortunately (as far as I know).

We currently control cache size via expiry dates, and those expiry values indirectly cause storage usage to be higher or lower. But the exact storage size is never guaranteed

We can create this "max storage" mechanism though. If I understand correctly, we are planning to do that as part of #1554, is my understanding accurate? If we are planning to address this there, maybe we can close this ticket and schedule work on #1554 for one of the upcoming sprints?

[alltheseas]

Agreed with @danieldaquino