Load testing

[josecelano]

I would like to add hundreds of random Torrens to the live demo.

• New console command to upload random torrents to the Index: seeder #453
• Upload torrents for a month and monitor the server.

I'm planning to upload:

3600 torrents/hour
86400 torrents/day
604800 torrents/week
2419200 torrents/month

[josecelano]

I have started uploading random torrents to the live demo:

2024-02-07T10:37:29.671005513+00:00 [seeder][INFO] Trying to login with username: admin ...
2024-02-07T10:37:30.280078305+00:00 [seeder][INFO] Logged as admin with account: admin 
2024-02-07T10:37:30.280152064+00:00 [seeder][INFO] Uploading 2419200 random torrents to the Torrust Index with a 1 seconds interval...
2024-02-07T10:37:30.280172397+00:00 [seeder][INFO] Uploading torrent #1 ...
2024-02-07T10:37:30.280199768+00:00 [seeder][INFO] Uploading torrent with uuid: 538f2a1b-d048-4976-8bd8-b1a1bf73d27e ...
2024-02-07T10:37:31.896728858+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":24,"info_hash":"882b8399ba5c5b45a0e6503efd311acef13f957d"}
2024-02-07T10:37:32.897125478+00:00 [seeder][INFO] Uploading torrent #2 ...
2024-02-07T10:37:32.897204089+00:00 [seeder][INFO] Uploading torrent with uuid: 45e94cef-1ce3-4e91-b06f-36a43f50821f ...
2024-02-07T10:37:35.015353750+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":25,"info_hash":"a1a7f87d0ae885d2690d796334da490bd4e5163c"}
2024-02-07T10:37:36.015561837+00:00 [seeder][INFO] Uploading torrent #3 ...
2024-02-07T10:37:36.015694115+00:00 [seeder][INFO] Uploading torrent with uuid: 0681510f-0c90-411e-9033-73c14d6806f2 ...
2024-02-07T10:37:37.288633248+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":26,"info_hash":"bd82002960d390a4edf16c0610dcbcb0f84e9ddb"}
2024-02-07T10:37:38.288803517+00:00 [seeder][INFO] Uploading torrent #4 ...
2024-02-07T10:37:38.288898443+00:00 [seeder][INFO] Uploading torrent with uuid: 05fb849c-7f72-4e85-b935-703f4088892a ...
2024-02-07T10:37:40.024003792+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":27,"info_hash":"f502332150ef445ffb808808c77ae1fa77bc8448"}
2024-02-07T10:37:41.024183655+00:00 [seeder][INFO] Uploading torrent #5 ...

[josecelano]

The server is quite busy today handling UDP announce requests:

In the last 12 days, the tracker has handled:

{
  "torrents": 38167,
  "seeders": 19020,
  "completed": 166956,
  "leechers": 28201,
  "tcp4_connections_handled": 6520,
  "tcp4_announces_handled": 6497,
  "tcp4_scrapes_handled": 23,
  "tcp6_connections_handled": 0,
  "tcp6_announces_handled": 0,
  "tcp6_scrapes_handled": 0,
  "udp4_connections_handled": 416488740,
  "udp4_announces_handled": 198269599,
  "udp4_scrapes_handled": 7227459,
  "udp6_connections_handled": 0,
  "udp6_announces_handled": 0,
  "udp6_scrapes_handled": 0
}

That is 191.232252122 UDP announce req/sec.

We are using the 6$/month droplet in Digital Ocean:

There are at least two problems:

Missing timeout in the seeder client

2024-02-07T16:54:30.067031801+00:00 [seeder][INFO] Uploading torrent #505 ...
2024-02-07T16:54:30.067125264+00:00 [seeder][INFO] Uploading torrent with uuid: 14b86e5c-994a-4b1a-af1d-a71857872f9b ...

It takes more than 1 second for the seeder to upload a new torrent.

Sometimes the seeder waits forever for the API response. We have to add a timeout to the API client. And retry after 5 seconds. Otherwise, you have to re-run it manually. However, we should find out why the seeder (API client) does not get any response later. Are we losing/discarding some requests?

Tracker Statistics Importer (cronjob) is too slow

The cronjob is importing 1 torrent per second with the current load:

index  | 2024-02-07T16:23:05.871624145+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1611 ...
index  | 2024-02-07T16:23:06.999765366+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2323 ...
index  | 2024-02-07T16:23:07.549943822+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1399 ...
index  | 2024-02-07T16:23:08.157716317+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 899 ...
index  | 2024-02-07T16:23:08.745064543+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2169 ...
index  | 2024-02-07T16:23:09.146923810+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 850 ...
index  | 2024-02-07T16:23:09.982634698+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2114 ...
index  | 2024-02-07T16:23:10.912697512+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1237 ...
index  | 2024-02-07T16:23:11.232916642+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2031 ...
index  | 2024-02-07T16:23:11.592363898+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2580 ...
index  | 2024-02-07T16:23:11.889599852+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2405 ...
index  | 2024-02-07T16:23:12.322081053+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2277 ...
index  | 2024-02-07T16:23:12.906672020+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1948 ...
index  | 2024-02-07T16:23:14.593848136+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 754 ...
index  | 2024-02-07T16:23:15.760567733+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 491 ...
index  | 2024-02-07T16:23:16.313846869+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 643 ...
index  | 2024-02-07T16:23:16.802825537+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2168 ...
index  | 2024-02-07T16:23:17.231140891+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 329 ...
index  | 2024-02-07T16:23:17.620177038+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1569 ...
index  | 2024-02-07T16:23:18.303492748+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 807 ...
index  | 2024-02-07T16:23:18.831055766+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 551 ...
index  | 2024-02-07T16:23:19.213513531+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 273 ...
index  | 2024-02-07T16:23:19.735776466+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 120 ...
index  | 2024-02-07T16:23:20.390742014+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 892 ...
index  | 2024-02-07T16:23:20.904715668+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2170 ...
index  | 2024-02-07T16:23:22.049670834+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2455 ...
index  | 2024-02-07T16:23:23.632835819+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2427 ...
index  | 2024-02-07T16:23:23.997367030+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 425 ...

That means we can only import 3600 torrents per hour. That's normal because it's a small droplet but the problem is sometimes it seems that the process is stopped. If I stop the seeder the cronjob continues working. If I re-start the seeder after a while the cronjob stops again. But it could be a coincidence.

See #428

I think I'm going to add more logs for HTTP Tracker and the API so that we can detect it there are:

• Some requests that are not processed.
• Some problems with the importer.

I think the problem with the importer could also be a missing timeout in the Tracker API client. I will open a new issue to track problems with timeouts as I did for the tracker.

cc @WarmBeer @da2ce7

[josecelano]

We have 27570 torrents in the live demo.

It seems the server is always busy finally.

cc @da2ce7 @WarmBeer

[josecelano]

I'm going to stop uploading more torrents. I think we have reached the server maximum load:

There are 27570 torrents. I guess 1000 torrents is the max acceptable load with the current level of UDP requests/sec. I should isolate the tests for the UDP tracker and for the Index importing the torrents from the tracker. Using a non-used UDP tracker to serve the Index. If we want to test only the performance of the statistics importer. But I only wanted to have a rough idea of how much load the smallest server can handle.