Merge #77: Command to migrate DB data from v1.0.0 to v2.0.0

5a7d875 feat: [#56] console command to import tracker stats for all torrents (Jose Celano)
19d054e refactor: [#56] rename test mods to follow prod mods (Jose Celano)
e8d984d refactor: [#56] rename destiny DB to target DB (Jose Celano)
b400962 fix: format (Jose Celano)
8b761c8 feat: [#56] keep category id in DB migration script (Jose Celano)
b29d4d7 fix: [#56] db migration for imported users (Jose Celano)
38fee53 test: [#56] new test for password verification (Jose Celano)
b9a8bf9 fix: [#56] remove comment (Jose Celano)
e1790f6 refactor: [#56] extract mods in upgrader (Jose Celano)
e23d948 refactor: remove duplication in tests (Jose Celano)
ee01e7b test: [#56] for torrent files table in upgrader (new case) (Jose Celano)
afffaef tests: [#56] for torrents files table in upgrader (Jose Celano)
82b84a3 refactor: [#56] extract test configuration (Jose Celano)
750969d refactor: [#56] rename methods (Jose Celano)
0063289 tests: [#56] for torrents info and announce urls tables in upgrader (Jose Celano)
cd95987 refactor: [#56] rename mod and variables (Jose Celano)
f0f581f tests: [#56] for torrents table in upgrader (Jose Celano)
eef980c tests: [#56] for tracker keys table in upgrader (Jose Celano)
8d74e66 tests: [#56] for users profile and auth tables in upgrader (Jose Celano)
0a58b6c fix: [#56] bio and avatar is user profile should be NULL for imported users (Jose Celano)
5d0def2 refactor: [#56] tests for upgrader (Jose Celano)
f993107 tests: [#56] for users table in upgrader (Jose Celano)
6188b10 refactor: extract mod sqlite_v1_0_0 in tests (Jose Celano)
44927e5 test: [#56] WIP. scaffolding to test upgrader command (Jose Celano)
7f0a7ea fix: open source db in read-only mode in upgarder (Jose Celano)
217fae2 feat: [#56] take source DB in upgrader command from args (Jose Celano)
aabc3ef feat: the upgrader command takes args (Jose Celano)
693994f feat: add new dependency text_colorizer (Jose Celano)
f620e05 fix: [#56] announce list has precedence over announce (Jose Celano)
309e141 fix: take torrent private flag from torrent file (Jose Celano)
72dc139 refactor: reformat sql queries (Jose Celano)
6bb4c53 refactor: extract struct TorrentRecordV2 (Jose Celano)
b9bf405 feat: [#56] improve command output (Jose Celano)
7152654 refactor: [#56] rename structs for DB records (Jose Celano)
99edf52 feat: imported users have importation date instead of registrataion date (Jose Celano)
21174d4 feat: [#56] trasnfer torrents (4/4 tables) from v1.0.0 to v2.0.0 (Jose Celano)
8bdf32f feat: [#56] trasnfer torrents (3/4 tables) from v1.0.0 to v2.0.0 (Jose Celano)
3fea6ea feat: [#56] trasnfer torrents (2/4 tables) from v1.0.0 to v2.0.0 (Jose Celano)
03e4bef feat: [#56] remove unused scripts and write basic upgrage guide (Jose Celano)
0b3aefa feat: [#56] transfer torrents (1/4 tables) from v1.0.0 to v2.0.0 (Jose Celano)
8d26faa fix: [#78] parsing keys from tracker (Jose Celano)
35f1e37 fix: [#56} default user registration date with time (Jose Celano)
dd949fa feat: [#56] transfer tracker keys from v1.0.0 to v2.0.0 (Jose Celano)
d9b4e87 feat: [#56] transfer user password from v1.0.0 to v2.0.0 (Jose Celano)
01921ed fix: [#56] triggering recompilation on migration changes (Jose Celano)
cf09283 docs: [#56] update README for integration tests (Jose Celano)
d1059f5 feat: [#56] trasnfer user data from v1.0.0 to v2.0.0 (Jose Celano)
d590972 refactor: [#56] move upgrader from main upgrade mod to specific version upgrader mod (Jose Celano)
996c7d1 refactor: [#56] rename command al dirs (Jose Celano)
b92fb08 feat: [#56] transfer categories from db v1.0.0 to v2.0.0 (Jose Celano)
7513df0 refactor: add scaffolding for database migration command (Jose Celano)
5d6dec0 refactor: allow adding more binaries (Jose Celano)
c3414da feat: add target dir to .gitignore (Jose Celano)

Pull request description:

  Depends on: #87

  I've created the basic scaffolding of the command. My plan is:

  - Migrate all the table records.
  - Insert torrents in DB from `uploads` dir.

  Potential problems are:

  - AUTOINCREMENTS
  - Foreign keys

  **UPDATE 2022-11-30**

  ### Tasks

  - [x] Transfer `torrent_categories`
  - [x] Transfer `torrent_user`
  - [x] Allow login for the transferred users using the Pbkdf2 for hashing the password
  - [x] Use date and time in `today_iso8601` function instead of only the date
  - [x] Transfer `torrent_tracker_keys`
  - [x] Remove unused content from dir `upgrades`. We do not need the MySQL stuff for this migration. There was no MySQL support for version `v1.0.0`
  - [x] Transfer `torrent_torrents`. See below the subtasks for this task
  - [x] Transfer `torrent_torrent_files`. It was not used in `v1.0.0`
  - [x] Make `registration_date` optional and leave it empty for imported users
  - [x] Add a new DB field `imported_date` to the `torrust_users` table with the importation date for user records imported with this command
  - [x] Refactor
  - [x] Add automated tests
  - [x] Intensive testing
  - [x] Write documentation explaining how to upgrade
  - [x] Rename `destiny` DB to `target` DB.

  ### Subtasks from issue #84

  This rework is needed because @ldpr found a bug in the current version that affects the upgrader. We have to fix that bug and update the "upgrader".
  - [x] Fix issue #84
  - [x] Rework: remove the `torrust_torrent_tracker_stats::tracker_url` column. **DISCARDED**.
  - [x] Populate table `torrust_torrent_tracker_stats` with one record per torrent.

  ### Transfer torrents subtasks

  - [x] Fill table `torrust_torrents`
  - [x] Fill table `torrust_torrent_files`
  - [x] Fill table `torrust_torrent_announce_urls`
  - [x] Fill table `torrust_torrent_info`

  ### Testing subtasks

  Assertions for integration test:

  - [x] `torrust_users` table
  - [x] `torrust_user_authentication` table
  - [x] `torrust_user_profiles` table
  - [x] `torrust_tracker_keys` table
  - [x] `torrust_torrents` table
  - [x] `torrust_torrent_files` table
  - [x] `torrust_torrent_info` table
  - [x] `torrust_torrent_announce_urls` table

  Some cases to test:

  - [x] Torrent with `pieces`.

  Using a single tracker URL or multiple URLs:

  - [x] Torrent with `announce` (single tracker url).
  - [x] Torrent with `announce_list` (multiple tracker URLs).

  Containing one file or multiple files:

  - [x] Torrent with one file.
  - [x] Torrent with multiple files.

  Add tests for the application behaviour changed:

  - [x] User can use have the password hashed with "argon2id" or a "pbkdf2-sha256". Add a test for the "verify_password" function.

  Things I'm not going to test:

  - [ ] Torrent with MD5 checksum?. I do not know how to include this MD5 checksum in the torrent file.
  - [ ] Torrent with `root_hash` ([BEP-0030](https://www.bittorrent.org/beps/bep_0030.html)).
  - [ ] The user registration date can be null for users that have been imported. We do not test it for MySQL since the upgrader is only for SQLite.

ACKs for top commit:
  josecelano:
    ACK 5a7d875
  da2ce7:
    ACK 5a7d875

Tree-SHA512: 0f79cb36bc8fac5f0166fa55d6addbbb1a4096c2f2e0e1d898b5b694bf16877cdc572a9336a69d0afd7da8c075edc21d49f6a5d640b187163269e59fe8f37138

.gitignore

@@ -1,4 +1,6 @@
 /.env
-/data.db*
 /config.toml
-/uploads/
+/data.db*
+/data_v2.db*
+/target
+/uploads/

Cargo.toml

@@ -3,6 +3,7 @@ name = "torrust-index-backend"
 version = "2.0.0-dev.1"
 authors = ["Mick van Dijke <[email protected]>", "Wesley Bijleveld <[email protected]>"]
 edition = "2021"
+default-run = "main"
 
 [profile.dev.package.sqlx-macros]
 opt-level = 3
@@ -33,3 +34,5 @@ tokio = {version = "1.13", features = ["macros", "io-util", "net", "time", "rt-m
 lettre = { version = "0.10.0-rc.3", features = ["builder", "tokio1", "tokio1-rustls-tls", "smtp-transport"]}
 sailfish = "0.4.0"
 regex = "1.6.0"
+pbkdf2 = "0.11.0"
+text-colorizer = "1.0.0"

build.rs

@@ -0,0 +1,5 @@
+// generated by `sqlx migrate build-script`
+fn main() {
+    // trigger recompilation when a new migration is added
+    println!("cargo:rerun-if-changed=migrations");
+}

migrations/mysql/20221109092556_torrust_user_date_registered_allow_null.sql

@@ -0,0 +1 @@
+ALTER TABLE torrust_users CHANGE date_registered date_registered DATETIME DEFAULT NULL

migrations/mysql/20221109095718_torrust_user_add_date_imported.sql

@@ -0,0 +1 @@
+ALTER TABLE torrust_users ADD COLUMN date_imported DATETIME DEFAULT NULL

migrations/sqlite3/20221109092556_torrust_user_date_registered_allow_null.sql

@@ -0,0 +1,11 @@
+CREATE TABLE IF NOT EXISTS torrust_users_new (
+    user_id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
+    date_registered TEXT DEFAULT NULL,
+    administrator BOOL NOT NULL DEFAULT FALSE
+);
+
+INSERT INTO torrust_users_new SELECT * FROM torrust_users;
+
+DROP TABLE torrust_users;
+
+ALTER TABLE torrust_users_new RENAME TO torrust_users

migrations/sqlite3/20221109095718_torrust_user_add_date_imported.sql

@@ -0,0 +1 @@
+ALTER TABLE torrust_users ADD COLUMN date_imported TEXT DEFAULT NULL

src/bin/import_tracker_statistics.rs

@@ -0,0 +1,10 @@
+//! Import Tracker Statistics command.
+//! It imports the number of seeders and leechers for all torrent from the linked tracker.
+//! You can execute it with: `cargo run --bin import_tracker_statistics`
+
+use torrust_index_backend::console::commands::import_tracker_statistics::run_importer;
+
+#[actix_web::main]
+async fn main() {
+    run_importer().await;
+}

src/bin/upgrade.rs

@@ -0,0 +1,10 @@
+//! Upgrade command.
+//! It updates the application from version v1.0.0 to v2.0.0.
+//! You can execute it with: `cargo run --bin upgrade ./data.db ./data_v2.db ./uploads`
+
+use torrust_index_backend::upgrades::from_v1_0_0_to_v2_0_0::upgrader::run_upgrader;
+
+#[actix_web::main]
+async fn main() {
+    run_upgrader().await;
+}

src/console/commands/import_tracker_statistics.rs

@@ -0,0 +1,86 @@
+//! It imports statistics for all torrents from the linked tracker.
+
+use std::env;
+use std::sync::Arc;
+
+use derive_more::{Display, Error};
+use text_colorizer::*;
+
+use crate::config::Configuration;
+use crate::databases::database::connect_database;
+use crate::tracker::TrackerService;
+
+const NUMBER_OF_ARGUMENTS: usize = 0;
+
+#[derive(Debug)]
+pub struct Arguments {}
+
+#[derive(Debug, Display, PartialEq, Error)]
+#[allow(dead_code)]
+pub enum ImportError {
+    #[display(fmt = "internal server error")]
+    WrongNumberOfArgumentsError,
+}
+
+fn parse_args() -> Result<Arguments, ImportError> {
+    let args: Vec<String> = env::args().skip(1).collect();
+
+    if args.len() != NUMBER_OF_ARGUMENTS {
+        eprintln!(
+            "{} wrong number of arguments: expected {}, got {}",
+            "Error".red().bold(),
+            NUMBER_OF_ARGUMENTS,
+            args.len()
+        );
+        print_usage();
+        return Err(ImportError::WrongNumberOfArgumentsError);
+    }
+
+    Ok(Arguments {})
+}
+
+fn print_usage() {
+    eprintln!(
+        "{} - imports torrents statistics from linked tracker.
+
+        cargo run --bin upgrade SOURCE_DB_FILE DESTINY_DB_FILE TORRENT_UPLOAD_DIR
+
+        For example:
+
+        cargo run --bin import_tracker_statistics
+
+        ",
+        "Upgrader".green()
+    );
+}
+
+pub async fn run_importer() {
+    import(&parse_args().unwrap()).await;
+}
+
+pub async fn import(_args: &Arguments) {
+    println!("Importing statistics from linked tracker ...");
+
+    let cfg = match Configuration::load_from_file().await {
+        Ok(config) => Arc::new(config),
+        Err(error) => {
+            panic!("{}", error)
+        }
+    };
+
+    let settings = cfg.settings.read().await;
+
+    let tracker_url = settings.tracker.url.clone();
+
+    eprintln!("Tracker url: {}", tracker_url.green());
+
+    let database = Arc::new(
+        connect_database(&settings.database.connect_url)
+            .await
+            .expect("Database error."),
+    );
+
+    let tracker_service = Arc::new(TrackerService::new(cfg.clone(), database.clone()));
+
+    tracker_service.update_torrents().await.unwrap();
+}

src/console/commands/mod.rs

@@ -0,0 +1 @@
+pub mod import_tracker_statistics;

src/console/mod.rs

@@ -0,0 +1 @@
+pub mod commands;

src/lib.rs

@@ -1,12 +1,14 @@
 pub mod auth;
 pub mod common;
 pub mod config;
+pub mod console;
 pub mod databases;
 pub mod errors;
 pub mod mailer;
 pub mod models;
 pub mod routes;
 pub mod tracker;
+pub mod upgrades;
 pub mod utils;
 
 trait AsCSV {

src/models/torrent_file.rs

@@ -39,6 +39,32 @@ pub struct TorrentInfo {
     pub root_hash: Option<String>,
 }
 
+impl TorrentInfo {
+    /// torrent file can only hold a pieces key or a root hash key:
+    /// http://www.bittorrent.org/beps/bep_0030.html
+    pub fn get_pieces_as_string(&self) -> String {
+        match &self.pieces {
+            None => "".to_string(),
+            Some(byte_buf) => bytes_to_hex(byte_buf.as_ref()),
+        }
+    }
+
+    pub fn get_root_hash_as_i64(&self) -> i64 {
+        match &self.root_hash {
+            None => 0i64,
+            Some(root_hash) => root_hash.parse::<i64>().unwrap(),
+        }
+    }
+
+    pub fn is_a_single_file_torrent(&self) -> bool {
+        self.length.is_some()
+    }
+
+    pub fn is_a_multiple_file_torrent(&self) -> bool {
+        self.files.is_some()
+    }
+}
+
 #[derive(PartialEq, Debug, Clone, Serialize, Deserialize)]
 pub struct Torrent {
     pub info: TorrentInfo, //
@@ -174,6 +200,27 @@ impl Torrent {
             }
         }
     }
+
+    pub fn announce_urls(&self) -> Vec<String> {
+        if self.announce_list.is_none() {
+            return vec![self.announce.clone().unwrap()];
+        }
+
+        self.announce_list
+            .clone()
+            .unwrap()
+            .into_iter()
+            .flatten()
+            .collect::<Vec<String>>()
+    }
+
+    pub fn is_a_single_file_torrent(&self) -> bool {
+        self.info.is_a_single_file_torrent()
+    }
+
+    pub fn is_a_multiple_file_torrent(&self) -> bool {
+        self.info.is_a_multiple_file_torrent()
+    }
 }
 
 #[derive(PartialEq, Eq, Debug, Clone, Serialize, Deserialize, sqlx::FromRow)]

src/models/tracker_key.rs

@@ -6,3 +6,15 @@ pub struct TrackerKey {
     pub key: String,
     pub valid_until: i64,
 }
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct NewTrackerKey {
+    pub key: String,
+    pub valid_until: Duration,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct Duration {
+    pub secs: i64,
+    pub nanos: i64,
+}

src/models/user.rs

@@ -3,7 +3,8 @@ use serde::{Deserialize, Serialize};
 #[derive(Debug, Serialize, Deserialize, Clone, sqlx::FromRow)]
 pub struct User {
     pub user_id: i64,
-    pub date_registered: String,
+    pub date_registered: Option<String>,
+    pub date_imported: Option<String>,
     pub administrator: bool,
 }
 
@@ -33,7 +34,8 @@ pub struct UserCompact {
 #[derive(Debug, Serialize, Deserialize, Clone, sqlx::FromRow)]
 pub struct UserFull {
     pub user_id: i64,
-    pub date_registered: String,
+    pub date_registered: Option<String>,
+    pub date_imported: Option<String>,
     pub administrator: bool,
     pub username: String,
     pub email: String,