Reduce size of all pages (36MB+) by removing apps

[markmhendrickson]

It appears we load JSON representing all apps on App.co for every page regardless of whether they're needed for its render. As apps have increased on the site, this JSON has ballooned in size, causing every page to have at least 36MB of weight.

We need to come up with a more efficient way for loading app data so we can get the page size down.

E.g.

"dataManager":"[]","props":{"pageProps":{"rankings":[{"category":"Utilities \u0026 Productivity","storageNetwork":"Gaia","authentication":"Blockstack","imgixImageUrl":"https://appco.imgix.net/apps/f6d63150-67da-4d9b-b8b1-ed3a2b0acd1c","id":2425,"name":"EasyGIF","categoryID":10,"website":"https://easygif.online/","blockchainID":null,"storageNetworkID":0,"authenticationID":0,"openSourceUrl":null,"registrationIsOpen":null,"trackingIsBlocked":null,"description":"Animated GIF Maker","twitterHandle":null,"gcsImagePath":"app.co/apps/f6d63150-67da-4d9b-b8b1-ed3a2b0acd1c","BTCAddress":"1LSPJVzmBiYdpNrT5UoYywfEFAD5aucXz3","isSubmittingOwnApp":true,"productionId":null,"imageUrl":"http://easygif.online/icon-192x192.jpg","createdAt":"2019-12-23T20:14:51.413Z","updatedAt":"2020-01-02T15:08:10.765Z","Slugs":[{"id":2346,"default":true,"value":"easygif","appId":2425,"createdAt":"2019-12-23T20:14:51.806Z","updatedAt":"2019-12-23T20:14:51.806Z"}],"slug":"easygif","rankings":[0.6482398638,1.528157229],"New Internet Labs Score":0.6482398638,"TryMyUI Score":1.528157229,"domain":"easygif.online","averageRanking":1.0206600059729438,"memoryRanking":1.0206600059729438,"usdRewards":20000,"formattedUsdRewards":"$20,000.00","btcRewards":0.5542008423852804,"formattedBtcRewards":"$0.55","payout":{"BTC":0.5542008423852804},"stxRewards":50000,"formattedSTXRewards":"50,000"},{"category":"Education \u0026 News","storageNetwork":"Gaia","authentication":"Blockstack","imgixImageUrl":"https://appco.imgix.net/apps/01acb08e-dc3f-4ce4-89aa-ab8f56b7cf5d","id":2096,"name":"Privus Reader","categoryID":2,"website":"https://privusreader.com/","blockchainID":null,"storageNetworkID":0,"authenticationID":0,"openSourceUrl":null,"registrationIsOpen":null,"trackingIsBlocked":null,"description":"A privacy-focused online reader","twitterHandle":null,"gcsImagePath":"app.co/apps/01acb08e-dc3f-4ce4-89aa-ab8f56b7cf5d","BTCAddress":"1LSPJVzmBiYdpNrT5UoYywfEFAD5aucXz3","isSubmittingOwnApp":true,"productionId":null,"imageUrl":"https://privusreader.com/icon-192x192.png","createdAt":"2019-10-30T18:36:05.525Z","updatedAt":"2019-10-30T20:14:20.723Z","Slugs":[{"id":1954,"default":true,"value":"privus-reader","appId":2096,"createdAt":"2019-10-30T18:36:06.061Z","updatedAt":"2019-10-30T18:36:06.061Z"}],"slug":"privus-reader","rankings":[1.496231738,0.6482398638,1.237492747],"Awario Score":1.496231738,"New Internet Labs Score":0.6482398638,"TryMyUI Score":1.237492747,"domain":"privusreader.com","averageRanking":1.0469218250923331,"memoryRanking":0.9980421577733692,"previousScore":0.8514031558164779,"usdRewards":19000,"formattedUsdRewards":"$19,000.00","btcRewards":0.5264908002660164,"formattedBtcRewards":"$0.53","payout":{"BTC":0.5264908002660164},"stxRewards":47500,"formattedSTXRewards":"47,500"},{"category":"Utilities \u0026 Productivity","storageNetwork":"Gaia","authentication":"Blockstack","imgixImageUrl":"https://appco.imgix.net/apps/6e669d29-c9ea-49b9-a469-013cd02abe86","id":2290,"name":"PDFstack","categoryID":10,"website":"https://pdfstack.online/","blockchainID":null,"storageNetworkID":0,"authenticationID":0,"openSourceUrl":null,"registrationIsOpen":null,"trackingIsBlocked":null,"description":"Merge \u0026 Split PDF files online. 100% Free \u0026 Privat","twitterHandle":null,"gcsImagePath":"app.co/apps/6e669d29-c9ea-49b9-a469-013cd02abe86","BTCAddress":"1CxQ6frzTVgo9xtKN25r6ZLrmNjHyqd8Fp","isSubmittingOwnApp":true,"productionId":null,"imageUrl":"https://pdfstack.online/icon-192x192.png","createdAt":"2019-11-29T22:26:30.836Z","updatedAt":"2019-11-30T02:31:30.707Z","Slugs":[{"id":2187,"default":true,"value":"pdfstack","appId":2290,"createdAt":"2019-11-29T22:26:31.229Z","updatedAt":"2019-11-29T22:26:31.229Z"}],"slug":"pdfstack","rankings":[1.448028761,0.6482398638,1.211031101],"Awario Score":1.448028761,"New Internet Labs Score":0.6482398638,"TryMyUI Score":1.211031101,"domain":"pdfstack.online","averageRanking":1.03631423141469,"memoryRanking":0.9950443720411319,"previousScore":0.8712347939204579,"usdRewards":18050,"formattedUsdRewards":"$18,050.00","btcRewards":0.5001662602527156,"formattedBtcRewards":"$0.50","payout":{"BTC":0.5001662602527156},"stxRewards":45125,"formattedSTXRewards":"45,125"},{"category":"Documents \u0026 Storage","storageNetwork":"Gaia","authentication":"Blockstack","imgixImageUrl":"https://appco.imgix.net/apps/409b27e0-5c04-48e0-b9a2-7a6e92cce4f6","id":2000,"name":"pDrive","categoryID":11,"website":"https://pdrive.co/","blockchainID":null,"storageNetworkID":0,"authenticationID":0,"openSourceUrl":null,"registrationIsOpen":null,"trackingIsBlocked":null,"description":"Private Encrypted Storage","twitterHandle":null,"gcsImagePath":"app.co/apps/409b27e0-5c04-48e0-b9a2-7a6e92cce4f6","BTCAddress":"3Eke2cB5z5G8yL1P6e4NwLyuCehQL8GBm5","isSubmittingOwnApp":true,"productionId":null,"imageUrl":"https://pdrive.co/icon-192x192.png","createdAt":"2019-09-27T20:43:40.174Z","updatedAt":"2019-09-27T20:53:32.709Z","Slugs":[{"id":1830,"default":true,"value":"pdrive","appId":2000,"createdAt":"2019-09-27T20:43:40.519Z","updatedAt":"2019-09-27T20:43:40.519Z"}],"slug":"pdrive","rankings":[1.251325442,0.6482398638,1.274474716],"Awario Score":1.251325442,"New Internet Labs Score":0.6482398638,"TryMyUI Score":1.274474716,"domain":"pdrive.co","averageRanking":1.0175621291785546,"memoryRanking":0.9896113599343803,"previousScore":0.9057590522018575,"usdRewards":17147.5,"formattedUsdRewards":"$17,147.50","btcRewards":0.4751579472400798,"formattedBtcRewards":"$0.48","payout":{"BTC":0.4751579472400798},"stxRewards":42868.75,"formattedSTXRewards":"42,869"},{"category":"Documents \u0026

[aulneau]

@markmhx I don't think this is technically correct. The redux store that has all the apps shows to be about 4kb. When looking at the network tab it does show that it's 36mb of "resources" but what I think this means is uncompressed/parsed javascript that is from the 4mb of assets transferred over the network.

For the JSON data to add anything meaningful to the 36mb you see there, there would have to be an insane amount of apps.

[kyranjamie]

Looking into the __NEXT_DATA__ json data that's included in the page source, I can see the full JSON representation for:

• 184 homepage apps
• 1209 apps

JSON.parse(document.querySelector('#__NEXT_DATA__').innerHTML).query.apps

As mark Mark alludes to, looks like this is loading the entire database into memory.

I think some serious refactoring is needed to untangle this, where each route will only load the data it needs. Hopefully, fetching all this data once load time isn't the cause of this issue.

[wileyj]

@kyranjamie that's what i've seen as well. no matter what's rendered on the page, the json is the entire DB of apps.
fetching the data initially causes some latency (first page load can take up to 5-6s), but the problem is that for any page fetched afterwards that 36MB file also has to be loaded.
As we add more dapps to the DB, I would expect this problem to only get worse over time.