Chat export 2000 MiB limit too low for attachments

[dali99]

Your use case

What would you like to do?

I would like to export my complete chat history but currently there's a limit of 2 000 MiB
This seems kind of like an arbitrary limit, though I understand that getting all the files and zipping them on the client side could get problematic for very big exports.
Unfortunately a 1:1 chat that is 4 years old can easilly have a larger footprint like thsi

How would you like to achieve it?

Allow more than 2 000 MiB of data to be exported

Have you considered any alternatives?

An alternative would be to allow paginating the export, I don't mind getting part1 part2 part3 zips each 2 GB in size.

Or if we could select date ranges like #19310 I could manually try split it up (though that would be a pain)

Additional context

No response

[germain-gg]

All the chat exporting is done client side and everything gets loaded in RAM if I remember correctly. The max file size limit is most likely to work around that.

Taking advantage of streams could be very useful to waive this limit, but browser support was too sporadic at the time of writing this functionality

[soda-pop-ice-cream]

I think devs added 2GB restriction to protect matrix.org from huge exports.
Also, I managed to export mine 4,5GB chat before restriction was added. At first export was failed because laptop had only 8GB RAM, but then i tried on other one, that had 16GB and that was barely enough. If you don't have that much RAM, at least use swap.
@dali99 if you really need that export you can install element-web docker v1.9.1 on some server, there was no 2GB size limit.

[t3chguy]

I think devs added 2GB restriction to protect matrix.org from huge exports.

Nope, it was added as anything larger often gets killed by the memory protections of modern browsers, even if you have far more ram the per-tab limit is far lower.

[Agent-Rogue]

for browsers, it's understandable since they wouldn't allow someone to export more than that anyway. however the native version (assuming electron doesn't have this limitation) shouldn't have this limitation.

[Szybet]

classic Matrix, such a stupid issue and still not solved after 2 years

[t3chguy]

@Szybet contributions welcome, the limit was imposed by the original external contributor of the feature based on experimental testing of common browser limitations which yield crashes

[Szybet]

Is it server side limited or client, because if server then I have tied hands

[t3chguy]

Client, there's no server-side export API, it just uses standard pagination APIs. It has to load the entire download into memory, hence the problem. Server can't help given end-to-end encryption.

[Szybet]

How do I build element desktop with a custom matrix-react-sdk

[t3chguy]

You build element-web with your matrix-react-sdk yarn link'ed in as per the README, then you symlink the resultant webapp into element-desktop/webapp and run yarn asar-webapp then build Element Desktop as normal ignoring the fetch instructions