Log all the original uploaded torrents

[josecelano]

Today, we discussed in our weekly meeting about losing information with the current index behaviour. We do not store the original uploaded torrent file data, and that could lead to two main problems:

• Lack of information for debugging if we found an error.
• It would not be possible to recover that information for future features.

Context

When you upload a torrent, the Index does not store all the information the torrent file has. We remove non-standard fields from the torrent file, and we store the standard fields in the database.

The most simple torrent could look like this in json format:

{
   "info": {
      "custom": "custom03",
      "length": 602515,
      "name": "mandelbrot_set_01",
      "piece length": 32768,
      "pieces": "<hex>8A 88 32 BE ED 05 5F AA C4 AF 4A 90 4B 9A BF 0D EC 83 42 1C 73 39 05 B8 D6 20 2C 1B D1 8A 53 28 1F B5 D4 23 0A 23 C8 DB AC C4 E6 6B 16 12 08 C7 A4 AD 64 45 70 ED 91 0D F1 38 E7 DF 0C 1A D0 C9 23 27 7C D1 F9 D4 E5 A1 5F F5 E5 A0 E4 9E FB B1 43 F5 4B AD 0E D4 9D CB 49 F7 E6 7B BA 30 5F AF F9 88 56 FB 45 9A B4 95 92 3E 2C 7F DA A6 D3 82 E7 63 A3 BB 4B 28 F3 57 C7 CB 7D 8C 06 E3 46 AB D7 E8 8E 8A 8C 9F C7 E6 C5 C5 64 82 ED 47 BB 2A F1 B7 3F A5 3C 5B 9C AF 43 EC 2A E1 08 68 9A 49 C8 BF 1B 07 AD BE E9 2D 7E BE 9C 18 7F 4C A1 97 0E 54 3A 18 94 0E 60 8D 5C 69 0E 41 46 0D 3C 9A 37 F6 81 62 4F 95 C0 73 92 CA 9A D5 A9 89 AC 8B 85 12 53 0B FB E2 96 26 3E 26 A6 5B 70 53 48 65 F3 6C 27 0F 6B BD 1C EE EB 1A 9D 5F 77 A8 D8 AF D8 14 82 4A E0 B4 62 BC F1 A5 F5 F2 C7 60 F8 38 C8 5B 0B A9 07 DD 86 FA C0 7B F0 26 D7 D1 9A 42 C3 1F 9F B9 59 83 10 62 41 E9 06 3C 6D A1 19 75 01 57 25 9E B7 FE DF 91 04 D4 51 4B 6D 44 02 8D 31 8E 84 26 95 0F 30 31 F0 2C 16 39 BD 53 1D CF D3 5E 3E 41 A9 1E 14 3F 73 24 AC 5E 9E FC 4D C5 70 45 0F 45 8B 9B 52 E6 D0 26 47 8F 43 08 9E 2A 7C C5 92 D5 86 36 FE 48 E9 B8 86 84 92 23 49 5B EE C4 31 B2 1D 10 75 8E 4C 07 84 8F</hex>"
   }
}

the_most_simple_torrent.zip

As you can see, it has a custom field in the info dictionary. As that is a non-standard field, we do not store it in the database, and that information is lost forever.

We want to store that information because:

• The infohash changes if we remove fields from the info dictionary. I'm already working on a new feature to store at least the original info-hash.
• In the future, some new features could depend on some custom fields that are not standard nowadays. If we store the whole torrent file data, we could implement those features for old data.
• We could change the actual Index behaviour, creating more than one torrent entry for all the torrents we are grouping now. Right now, a torrent entry represents the "canonical torrent". You cannot upload a torrent whose infohash, after removing the non-standard fields (canonical infohash), is the same as a previously uploaded one.

We can add the solution discussed below or add more solutions.

cc @da2ce7 @mario-nt @grmbyrn

[josecelano]

Solution 1

We can store the original torrent file in a log folder: storage/index/log/uploaded/torrents, for example.

We can use a hash of the torrent file as the file name: aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d.torrent

Ideally, the frontend should calculate the hash so we do not have to consume the server CPU on that task.

We could also use a feature flag and let the admin decide if they want this feature enabled or not.

These would lead to some other advantages. If we change the frontend we could split the upload process in two steps (for the user or only internally for the API requests). In the first step, we calculate the hash and then we try to make a pre-request to add the new torrent to the index. The pre-request can only contain the metadata info: title, description, category, tags, and the hash of the real torrent file. The second request is to upload only the binary file or, again, the whole metadata plus the binary file. We can start storing the hash in the Index and avoid duplicate uploads for the same torrent file. Although this feature should be considered if we store the original torrent file not only as a log artefact but as a first-class app entity.

[da2ce7]

Ideally, the frontend should calculate the hash so we do not have to consume the server CPU on that task.

The server needs to verify that the torrent hash matches; we cannot trust the user to be honest.

[da2ce7]

We can start storing the hash in the Index and avoid duplicate uploads for the same torrent file. Although this feature should be considered if we store the original torrent file not only as a log artefact but as a first-class app entity.

We should store the hash of the uploaded torrent file in any case. Even if we do not keep the torrent files themselves. There should be a table, like the info-hash > canonical info-hash. We can have a new table: torrent-hash > info-hash.