From dde2fb3afaf64c2f62de65380ae79ee480a82e7e Mon Sep 17 00:00:00 2001 From: Jan Fabian Radon Date: Thu, 22 Sep 2022 13:33:49 +0200 Subject: [PATCH] Feature/342 object search (#343) * Develop (#331) * #328: fix copy move (#330) * Develop (#334) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * Bugfix/283 virtual mailbox fetch metadata (#284) * Develop (#282) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * #283: fetch metadata for mails in virtual mailbox * #283: fix metadata date.saved. date.received virtual mailbox * #283: version * Feature/286 use guid from UUID string (#287) * #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing * version 0.0.25 preparations * #286: build issue * merge * version * #286: prefere cached mail guid. * #286: determine if uuid has - hyphon, if true preseve it. * fix non void return * Feature/289 GitHub actions (#290) * #289: build * #289: build plugin. * #289: submodules * submodules * upgrade git * clean outdated repos * Bugfix/UUID record format (#293) * bugfix initialisaction rados_mail->deprecated_uid * rados_mail creation and default value check * #295: update index after full object has been moved (metadata) (#296) * Bugfix/298 295 bugfixes (#300) * #289: fix force-resync (always use INBOX) #295: rbox_mail_set_expunge (play save) * #298: move ceph objects the standard way. * clean up log. * #298: log warning if client connection times out. * Feature/302 retry expunge connection timeout (#303) * #302: retry in case of ceph connection timeout * rpm * #302: max_retry = 10 * max 10 retry, in case object not available, fail with error * #304: use last know uid to restore email flags (#305) * #306: re-assign unreferenced mail objects to inbox (#309) * #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311) * version: 0.0.31 * #313: fix imap append crash (#314) * version 0.0.32 * Bugfix/316 mailbox save (#317) * #316: imap append (remove index entry twice in case of error) * #316: changed storage for big attachments. (sync write) * #316: operate api * #316: fix operate call * #316: release preparations * #316: fixed some legacy unit tests * 316: review results * #316: fix * Bugfix/test output stream check (#323) * init output stream and free mailbuffer * build * extracted write chunks * mailsize -1; * fix * debug messages * init stream any case * added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes * missing mock class * version * disabled (temporarily storage tests) * disabled storage tests( api change) * Feature/alternative save methods (#326) * #322: alternative save methods * version * Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321) * #319: automatically add lost objects to inbox (if no mailbox guid exist) * #319: version 0.0.34 * catch error in case mailbox does not have obox header: e.v. Virtual Mailbox * check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin) * version * #328: fix copy move (#330) * Bugfix/332 quota and move (#333) * 291: failed save mock test * 291: fixed most of the tests * 332 fix notify messages number and type * Develop (#336) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * Bugfix/283 virtual mailbox fetch metadata (#284) * Develop (#282) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * #283: fetch metadata for mails in virtual mailbox * #283: fix metadata date.saved. date.received virtual mailbox * #283: version * Feature/286 use guid from UUID string (#287) * #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing * version 0.0.25 preparations * #286: build issue * merge * version * #286: prefere cached mail guid. * #286: determine if uuid has - hyphon, if true preseve it. * fix non void return * Feature/289 GitHub actions (#290) * #289: build * #289: build plugin. * #289: submodules * submodules * upgrade git * clean outdated repos * Bugfix/UUID record format (#293) * bugfix initialisaction rados_mail->deprecated_uid * rados_mail creation and default value check * #295: update index after full object has been moved (metadata) (#296) * Bugfix/298 295 bugfixes (#300) * #289: fix force-resync (always use INBOX) #295: rbox_mail_set_expunge (play save) * #298: move ceph objects the standard way. * clean up log. * #298: log warning if client connection times out. * Feature/302 retry expunge connection timeout (#303) * #302: retry in case of ceph connection timeout * rpm * #302: max_retry = 10 * max 10 retry, in case object not available, fail with error * #304: use last know uid to restore email flags (#305) * #306: re-assign unreferenced mail objects to inbox (#309) * #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311) * version: 0.0.31 * #313: fix imap append crash (#314) * version 0.0.32 * Bugfix/316 mailbox save (#317) * #316: imap append (remove index entry twice in case of error) * #316: changed storage for big attachments. (sync write) * #316: operate api * #316: fix operate call * #316: release preparations * #316: fixed some legacy unit tests * 316: review results * #316: fix * Bugfix/test output stream check (#323) * init output stream and free mailbuffer * build * extracted write chunks * mailsize -1; * fix * debug messages * init stream any case * added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes * missing mock class * version * disabled (temporarily storage tests) * disabled storage tests( api change) * Feature/alternative save methods (#326) * #322: alternative save methods * version * Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321) * #319: automatically add lost objects to inbox (if no mailbox guid exist) * #319: version 0.0.34 * catch error in case mailbox does not have obox header: e.v. Virtual Mailbox * check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin) * version * #328: fix copy move (#330) * Bugfix/332 quota and move (#333) * 291: failed save mock test * 291: fixed most of the tests * 332 fix notify messages number and type * #332: fix append additional error mssage in case connection can't be opend (#335) * Develop (#341) * Loglevel Debug is sufficient as it gets retried (#337) * Bugfix/339 rados config timeout (#340) * 339 retry 10 times with random wait. retry read for simple read e.g. rados_config. fail with assert in case we can't find rados_config * #342: simple multithreading object search * #342: version 0.0.40 Co-authored-by: Ewald Dieterich --- CHANGELOG.md | 8 + configure.ac | 2 +- rpm/dovecot-ceph-plugin.spec | 2 +- src/librmb/rados-ceph-config.h | 2 +- src/librmb/rados-cluster-impl.cpp | 53 ++++- src/librmb/rados-cluster-impl.h | 3 + src/librmb/rados-cluster.h | 7 + src/librmb/rados-dovecot-ceph-cfg-impl.h | 2 + src/librmb/rados-dovecot-ceph-cfg.h | 3 + src/librmb/rados-dovecot-config.cpp | 13 +- src/librmb/rados-dovecot-config.h | 6 + src/librmb/rados-storage-impl.cpp | 58 ++++- src/librmb/rados-storage-impl.h | 6 + src/librmb/rados-storage.h | 6 + src/librmb/rados-util.cpp | 72 ++++++ src/librmb/rados-util.h | 16 ++ src/storage-rbox/rbox-sync-rebuild.cpp | 47 +++- src/storage-rbox/rbox-sync-rebuild.h | 5 +- src/tests/Makefile.am | 6 + src/tests/mocks/mock_test.h | 9 + .../storage-mock-rbox/test_repair_rbox.cpp | 214 ++++++++++++++++++ 21 files changed, 517 insertions(+), 23 deletions(-) create mode 100644 src/tests/storage-mock-rbox/test_repair_rbox.cpp diff --git a/CHANGELOG.md b/CHANGELOG.md index 71f992ad..6c52d7ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Change Log +## [0.0.40](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.39) (2022-09-22) +- #342 multithreading object search for doveadm force-resync (feature toggle) + new config params: + # search method default = 0 | 1 multithreading + rbox_object_search_method=1 + # number of threads to use in case of search_method=1 + rbox_object_search_threads=4 + ## [0.0.39](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.39) (2022-08-25) - #339 fail with assert if rados_config cannot be found due to network/connection issue retry ceph read operations / read / xattr with timeout diff --git a/configure.ac b/configure.ac index f09b4af8..b8fb59ef 100644 --- a/configure.ac +++ b/configure.ac @@ -9,7 +9,7 @@ AC_PREREQ([2.59]) -AC_INIT([dovecot-ceph-plugin], [0.0.39], [https://github.com/ceph-dovecot/dovecot-ceph-plugin/issues/new], ,[https://github.com/ceph-dovecot/dovecot-ceph-plugin]) +AC_INIT([dovecot-ceph-plugin], [0.0.40], [https://github.com/ceph-dovecot/dovecot-ceph-plugin/issues/new], ,[https://github.com/ceph-dovecot/dovecot-ceph-plugin]) AC_CONFIG_AUX_DIR([.]) diff --git a/rpm/dovecot-ceph-plugin.spec b/rpm/dovecot-ceph-plugin.spec index e10c1939..aa5f1b3b 100644 --- a/rpm/dovecot-ceph-plugin.spec +++ b/rpm/dovecot-ceph-plugin.spec @@ -15,7 +15,7 @@ Name: dovecot-ceph-plugin Summary: Dovecot Ceph RADOS plugins -Version: 0.0.39 +Version: 0.0.40 Release: 0%{?dist} URL: https://github.com/ceph-dovecot/dovecot-ceph-plugin diff --git a/src/librmb/rados-ceph-config.h b/src/librmb/rados-ceph-config.h index a1a55959..e9cbccde 100644 --- a/src/librmb/rados-ceph-config.h +++ b/src/librmb/rados-ceph-config.h @@ -58,7 +58,7 @@ class RadosCephConfig { void set_update_attributes(const std::string &update_attributes_) { config.set_update_attributes(update_attributes_); } - + void update_mail_attribute(const char *value) { config.update_mail_attribute(value); } void update_updateable_attribute(const char *value) { config.update_updateable_attribute(value); } diff --git a/src/librmb/rados-cluster-impl.cpp b/src/librmb/rados-cluster-impl.cpp index 1a68bfb7..00820073 100644 --- a/src/librmb/rados-cluster-impl.cpp +++ b/src/librmb/rados-cluster-impl.cpp @@ -17,7 +17,7 @@ #include "rados-dictionary-impl.h" #include "rados-storage-impl.h" - +#include "rados-util.h" using std::list; using std::pair; using std::string; @@ -72,6 +72,57 @@ int RadosClusterImpl::init(const std::string &clustername, const std::string &ra return ret; } + +std::vector RadosClusterImpl::list_pgs_for_pool(std::string &pool_name) { + std::cout << " ola " << RadosClusterImpl::cluster << std::endl; + + if(is_connected()){ + std::cout << " is connected YES" << std::endl; + }else{ + std::cout << " is connected NO" << std::endl; + connect(); + } + + const string pool = "mail_storage"; + const string cmd = + "{" + "\"prefix\": \"pg ls-by-pool\", " + "\"poolstr\": \"" + pool + "\"" + "}"; + + std::cout << "cmd: " << cmd << std::endl; + + librados::bufferlist inbl; + librados::bufferlist outbl; + int res = RadosClusterImpl::cluster->mon_command(cmd, inbl, &outbl, nullptr); + std::cout << "inbl command " << inbl < list = RadosUtils::extractPgs(std::string(outbl.c_str())); + + for (auto const &token: list) { + std::cout << token << std::endl; + } + return list; +} + +std::map> RadosClusterImpl::list_pgs_osd_for_pool(std::string &pool_name) { + + if(!is_connected()){ + connect(); + } + + const string cmd = + "{" + "\"prefix\": \"pg ls-by-pool\", " + "\"poolstr\": \"" + pool_name + "\"" + "}"; + + librados::bufferlist inbl; + librados::bufferlist outbl; + RadosClusterImpl::cluster->mon_command(cmd, inbl, &outbl, nullptr); + return RadosUtils::extractPgAndPrimaryOsd(std::string(outbl.c_str())); +} int RadosClusterImpl::initialize() { int ret = 0; diff --git a/src/librmb/rados-cluster-impl.h b/src/librmb/rados-cluster-impl.h index ca9678f3..8786e023 100644 --- a/src/librmb/rados-cluster-impl.h +++ b/src/librmb/rados-cluster-impl.h @@ -40,6 +40,9 @@ class RadosClusterImpl : public RadosCluster { librados::Rados &get_cluster() { return *cluster; } void set_config_option(const char *option, const char *value); + std::vector list_pgs_for_pool(std::string &pool_name) override; + std::map> list_pgs_osd_for_pool(std::string &pool_name) override; + private: int initialize(); diff --git a/src/librmb/rados-cluster.h b/src/librmb/rados-cluster.h index dac4c59a..60dbb79a 100644 --- a/src/librmb/rados-cluster.h +++ b/src/librmb/rados-cluster.h @@ -76,6 +76,13 @@ class RadosCluster { * @return true if connected */ virtual bool is_connected() = 0; + + /*! get placement groups for mailbox storage pool + */ + virtual std::vector list_pgs_for_pool(std::string &pool_name) = 0; + virtual std::map> list_pgs_osd_for_pool(std::string &pool_name) = 0; + + }; } // namespace librmb diff --git a/src/librmb/rados-dovecot-ceph-cfg-impl.h b/src/librmb/rados-dovecot-ceph-cfg-impl.h index 811bc197..25a3e279 100644 --- a/src/librmb/rados-dovecot-ceph-cfg-impl.h +++ b/src/librmb/rados-dovecot-ceph-cfg-impl.h @@ -58,6 +58,8 @@ class RadosDovecotCephCfgImpl : public RadosDovecotCephCfg { rados_cfg.set_cfg_object_name(dovecot_cfg.get_rbox_cfg_object_name()); } } + int get_object_search_method() override { return std::stoi(dovecot_cfg.get_object_search_method()); } + int get_object_search_threads() override { return std::stoi(dovecot_cfg.get_object_search_threads()); } void set_rbox_cfg_object_name(const std::string &value) override { dovecot_cfg.set_rbox_cfg_object_name(value); } diff --git a/src/librmb/rados-dovecot-ceph-cfg.h b/src/librmb/rados-dovecot-ceph-cfg.h index b7dd7a08..c73bf0c4 100644 --- a/src/librmb/rados-dovecot-ceph-cfg.h +++ b/src/librmb/rados-dovecot-ceph-cfg.h @@ -44,6 +44,9 @@ class RadosDovecotCephCfg { virtual int get_chunk_size() = 0; virtual int get_write_method() = 0; + virtual int get_object_search_method() = 0; + virtual int get_object_search_threads() = 0; + virtual const std::string &get_pool_name_metadata_key() = 0; virtual const std::string &get_update_attributes_key() = 0; virtual const std::string &get_mail_attributes_key() = 0; diff --git a/src/librmb/rados-dovecot-config.cpp b/src/librmb/rados-dovecot-config.cpp index 5233db4b..874df8eb 100644 --- a/src/librmb/rados-dovecot-config.cpp +++ b/src/librmb/rados-dovecot-config.cpp @@ -29,7 +29,9 @@ RadosConfig::RadosConfig() rbox_ceph_aio_wait_for_safe_and_cb("rbox_ceph_aio_wait_for_safe_and_cb"), rbox_ceph_write_chunks("rbox_ceph_write_chunks"), rbox_chunk_size("rbox_chunk_size"), - rbox_write_method("rbox_write_method") { + rbox_write_method("rbox_write_method"), + rbox_object_search_method("rbox_object_search_method"), + rbox_object_search_threads("rbox_object_search_threads") { config[pool_name] = "mail_storage"; @@ -43,6 +45,9 @@ RadosConfig::RadosConfig() config[rbox_ceph_write_chunks] = "false"; config[rbox_chunk_size] = "10240"; config[rbox_write_method] = "0"; + config[rbox_object_search_method] = "0"; + config[rbox_object_search_threads] = "4"; + is_valid = false; } @@ -79,8 +84,10 @@ std::string RadosConfig::to_string() { ss << " " << rbox_ceph_aio_wait_for_safe_and_cb << "=" << config[rbox_ceph_aio_wait_for_safe_and_cb] << std::endl; ss << " " << rbox_ceph_write_chunks << "=" << config[rbox_ceph_write_chunks] << std::endl; ss << " " << rbox_write_method << "=" << config[rbox_write_method] << std::endl; - ss << " " << rbox_chunk_size << "=" << config[rbox_chunk_size] - << std::endl; + ss << " " << rbox_chunk_size << "=" << config[rbox_chunk_size] << std::endl; + ss << " " << rbox_object_search_method << "=" << config[rbox_object_search_method] << std::endl; + ss << " " << rbox_object_search_threads << "=" << config[rbox_object_search_threads] << std::endl; + return ss.str(); } diff --git a/src/librmb/rados-dovecot-config.h b/src/librmb/rados-dovecot-config.h index c469d2e8..89900a5e 100644 --- a/src/librmb/rados-dovecot-config.h +++ b/src/librmb/rados-dovecot-config.h @@ -49,6 +49,10 @@ class RadosConfig { const std::string &get_rbox_cluster_name() { return config[rbox_cluster_name]; } const std::string &get_rados_username() { return config[rados_username]; } + + const std::string &get_object_search_method() { return config[rbox_object_search_method]; } + const std::string &get_object_search_threads() { return config[rbox_object_search_threads]; } + void update_metadata(const std::string &key, const char *value_); bool is_ceph_posix_bugfix_enabled() { return config[bugfix_cephfs_posix_hardlinks].compare("true") == 0 ? true : false; @@ -87,6 +91,8 @@ class RadosConfig { std::string rbox_ceph_write_chunks; std::string rbox_chunk_size; std::string rbox_write_method; + std::string rbox_object_search_method; + std::string rbox_object_search_threads; bool is_valid; }; diff --git a/src/librmb/rados-storage-impl.cpp b/src/librmb/rados-storage-impl.cpp index 1dc9b719..5525bf36 100644 --- a/src/librmb/rados-storage-impl.cpp +++ b/src/librmb/rados-storage-impl.cpp @@ -16,8 +16,13 @@ #include #include #include +#include +#include + +#include "rados-util.h" #include + #include "encoding.h" #include "limits.h" @@ -173,7 +178,6 @@ librados::NObjectIterator RadosStorageImpl::find_mails(const RadosMetadata *attr } if (attr != nullptr) { - // int hashpos = get_io_ctx().get_object_hash_position("t1_u"); std::string filter_name = PLAIN_FILTER_NAME; ceph::bufferlist filter_bl; @@ -186,7 +190,57 @@ librados::NObjectIterator RadosStorageImpl::find_mails(const RadosMetadata *attr return get_io_ctx().nobjects_begin(); } } - +/** + * POC Implementation: + * + * see in prod how it behaves. + * + **/ +std::set RadosStorageImpl::find_mails_async(const RadosMetadata *attr, + std::string &pool_name, + int num_threads){ + + std::set oid_list; + std::mutex oid_list_mutex; + + // Define a Lambda Expression + auto f = [](const std::vector &list, std::mutex &oid_mutex, std::set &oids, librados::IoCtx *io_ctx) { + + std::lock_guard guard(oid_mutex); + for (auto const &pg: list) { + uint64_t ppool; + uint32_t pseed; + int r = sscanf(pg.c_str(), "%llu.%x", (long long unsigned *)&ppool, &pseed); + + librados::NObjectIterator iter= io_ctx->nobjects_begin(pseed); + + while (iter != librados::NObjectIterator::__EndObjectIterator) { + std::string oid = iter->get_oid(); + oids.insert(oid); + iter++; + } + } + }; + + //std::string pool_mame = "mail_storage"; + std::map> osd_pg_map = cluster->list_pgs_osd_for_pool(pool_name); + std::vector threads; + + for (const auto& x : osd_pg_map) + { + if(threads.size() == num_threads){ + for (auto const &thread: threads) { + thread.join(); + } + threads.clear(); + } + threads.push_back(std::thread(f, std::ref(x.second),std::ref(oid_list_mutex),std::ref(oid_list), &get_io_ctx())); + } + for (auto const &thread: threads) { + thread.join(); + } + return oid_list; +} librados::IoCtx &RadosStorageImpl::get_io_ctx() { return io_ctx; } int RadosStorageImpl::open_connection(const std::string &poolname, const std::string &clustername, diff --git a/src/librmb/rados-storage-impl.h b/src/librmb/rados-storage-impl.h index 3712f217..8984524f 100644 --- a/src/librmb/rados-storage-impl.h +++ b/src/librmb/rados-storage-impl.h @@ -18,6 +18,8 @@ #include #include #include +#include + #include #include @@ -50,6 +52,9 @@ class RadosStorageImpl : public RadosStorage { int aio_operate(librados::IoCtx *io_ctx_, const std::string &oid, librados::AioCompletion *c, librados::ObjectWriteOperation *op) override; librados::NObjectIterator find_mails(const RadosMetadata *attr) override; + + std::set find_mails_async(const RadosMetadata *attr, std::string &pool_name, int num_threads) override; + int open_connection(const std::string &poolname) override; int open_connection(const std::string &poolname, const std::string &clustername, const std::string &rados_username) override; @@ -72,6 +77,7 @@ class RadosStorageImpl : public RadosStorage { void free_rados_mail(librmb::RadosMail *mail) override; + private: int create_connection(const std::string &poolname); diff --git a/src/librmb/rados-storage.h b/src/librmb/rados-storage.h index e723e772..f5234026 100644 --- a/src/librmb/rados-storage.h +++ b/src/librmb/rados-storage.h @@ -104,6 +104,11 @@ class RadosStorage { * * @return object iterator or librados::NObjectIterator::__EndObjectIterator */ virtual librados::NObjectIterator find_mails(const RadosMetadata *attr) = 0; + + + virtual std::set find_mails_async(const RadosMetadata *attr, std::string &pool_name, int num_threads) = 0; + + /*! open the rados connections with default cluster and username * @param[in] poolname the poolname to connect to, in case this one does not exists, it will be created. * */ @@ -199,6 +204,7 @@ class RadosStorage { * */ virtual void free_rados_mail(librmb::RadosMail *mail) = 0; + }; } // namespace librmb diff --git a/src/librmb/rados-util.cpp b/src/librmb/rados-util.cpp index b2df83d0..be637ab6 100644 --- a/src/librmb/rados-util.cpp +++ b/src/librmb/rados-util.cpp @@ -20,6 +20,8 @@ #include #include #include +#include +#include #include "encoding.h" namespace librmb { @@ -292,4 +294,74 @@ int RadosUtils::copy_to_alt(std::string &src_oid, std::string &dest_oid, RadosSt return success ? 0 : 1; } +static std::vector RadosUtils::extractPgs(const std::string& str) +{ + std::vector tokens; + + std::stringstream ss(str); + std::string token; + while (std::getline(ss, token, '\n')) { + std::string pgs = token.substr(0, 10); //take first 10 chars for pgids + // trim the result. + pgs.erase(std::remove_if(pgs.begin(), pgs.end(), ::isspace), pgs.end()); + tokens.push_back(pgs); + } + //skip first line (header) + tokens.erase(tokens.begin()); + //remove last element (footer) + tokens.pop_back(); + + return tokens; +} + +static std::map> RadosUtils::extractPgAndPrimaryOsd(const std::string& str) +{ + std::map> tokens; + + std::stringstream ss(str); + std::string token; + bool first_line = true; + while (std::getline(ss, token, '\n')) { + if(first_line){ + first_line = false; + continue; + } + std::vector line = split(token,' '); + if(line.size() < 14){ + continue; + } + std::string tmp_primary_osd = split(line[13],',')[0]; + std::string primary_osd = tmp_primary_osd.erase(0,1); + std::string pgs = line[0]; + + auto it = tokens.find(primary_osd); + if(it!=tokens.end()){ + tokens[primary_osd].push_back(pgs); + }else{ + std::vector t; + t.push_back(pgs); + tokens.insert({primary_osd,t}); + } + + } + + return tokens; +} + +static std::vector RadosUtils::split(std::string str_to_split, char delimiter) { + std::vector tokens; + std::stringstream stream(str_to_split); + std::string token; + + while(getline(stream, token, delimiter)) { + token.erase(std::remove_if(token.begin(), token.end(), ::isspace), token.end()); + if(token.length() > 0) { + tokens.push_back(token); + } + + } + + return tokens; +} + } // namespace librmb diff --git a/src/librmb/rados-util.h b/src/librmb/rados-util.h index 566d7c26..e6b019dc 100644 --- a/src/librmb/rados-util.h +++ b/src/librmb/rados-util.h @@ -16,6 +16,11 @@ #include #include +#include +#include +#include +#include + #include #include #include @@ -180,6 +185,17 @@ class RadosUtils { * @return the metadata value */ static void get_metadata(rbox_metadata_key key, std::map *metadata, char **value); + + + /** + * POC Implemnentation to extract pgs and primary osds from mon_command output! + **/ + static std::vector extractPgs(const std::string& str); + + static std::map> extractPgAndPrimaryOsd(const std::string& str); + + static std::vector split(std::string str_to_split, char delimiter); + }; } // namespace librmb diff --git a/src/storage-rbox/rbox-sync-rebuild.cpp b/src/storage-rbox/rbox-sync-rebuild.cpp index 7d6a998e..6ae4eea8 100644 --- a/src/storage-rbox/rbox-sync-rebuild.cpp +++ b/src/storage-rbox/rbox-sync-rebuild.cpp @@ -102,14 +102,19 @@ int rbox_sync_add_object(struct index_rebuild_context *ctx, const std::string &o std::map> load_rados_mail_metadata( bool alt_storage, struct rbox_storage *r_storage, - librados::NObjectIterator &iter) { + std::set &mail_list) { std::map> rados_mails; - while (iter != librados::NObjectIterator::__EndObjectIterator) { + std::set::iterator it; + + for(it=mail_list.begin(); it!=mail_list.end(); ++it){ + + //while (iter != librados::NObjectIterator::__EndObjectIterator) { librmb::RadosMail mail_object; - mail_object.set_oid((*iter).get_oid()); + + mail_object.set_oid((*it)); int load_metadata_ret; if (alt_storage) { @@ -121,7 +126,7 @@ std::map> load_rados_mail_metadata( if (load_metadata_ret < 0 || !librmb::RadosUtils::validate_metadata(mail_object.get_metadata())) { i_warning("metadata for object : %s is not valid, skipping object ", mail_object.get_oid()->c_str()); - ++iter; + //++iter; continue; } @@ -141,7 +146,8 @@ std::map> load_rados_mail_metadata( rados_mails[mailbox_guid]= list_mail_objects; } } - ++iter; + + // ++iter; } return rados_mails; } @@ -416,6 +422,9 @@ int repair_namespace(struct mail_namespace *ns, bool force, struct rbox_storage const struct mailbox_info *info; int ret = 0; + + + iter = mailbox_list_iter_init(ns->list, "*", static_cast(MAILBOX_LIST_ITER_RAW_LIST | MAILBOX_LIST_ITER_RETURN_NO_FLAGS)); while ((info = mailbox_list_iter_next(iter)) != NULL) { @@ -424,7 +433,7 @@ int repair_namespace(struct mail_namespace *ns, bool force, struct rbox_storage struct mailbox *box = mailbox_alloc(ns->list, info->vname, MAILBOX_FLAG_SAVEONLY); if (box->storage != &r_storage->storage || - box->virtual_vfuncs != NULL) { + box->virtual_vfuncs != NULL) { /* the namespace has multiple storages. or is virtual box */ mailbox_free(&box); return 0; @@ -443,11 +452,29 @@ int repair_namespace(struct mail_namespace *ns, bool force, struct rbox_storage FUNC_END(); return -1; } - i_info("Ceph connection established using namespace: %s",r_storage->s->get_namespace().c_str()); - i_info("Loading mails... "); - librados::NObjectIterator *iter_guid = new librados::NObjectIterator(r_storage->s->find_mails(nullptr)); + + + std::set mail_list; + std::string pool_name = r_storage->s->get_pool_name(); + if( r_storage->config->get_object_search_method() == 1) { + mail_list = r_storage->s->find_mails_async(nullptr, + pool_name, + r_storage->config->get_object_search_threads()); + i_info("multithreading done"); + }else{ + i_info("Ceph connection established using namespace: %s",r_storage->s->get_namespace().c_str()); + i_info("Loading mails... "); + librados::NObjectIterator iter_guid = r_storage->s->find_mails(nullptr); + while (iter_guid != librados::NObjectIterator::__EndObjectIterator) { + mail_list.insert((*iter_guid).get_oid()); + i_debug("iter mail list %s",(*iter_guid).get_oid().c_str()); + iter_guid++; + } + + } + i_info("Loading mail metadata..."); - rados_mails = load_rados_mail_metadata(false,r_storage,*iter_guid); + rados_mails = load_rados_mail_metadata(false,r_storage, mail_list); i_info("Mails completely loaded "); std::map>::iterator it; for(it=rados_mails.begin(); it!=rados_mails.end(); ++it){ diff --git a/src/storage-rbox/rbox-sync-rebuild.h b/src/storage-rbox/rbox-sync-rebuild.h index 81dc26a5..16bed509 100644 --- a/src/storage-rbox/rbox-sync-rebuild.h +++ b/src/storage-rbox/rbox-sync-rebuild.h @@ -43,13 +43,10 @@ extern int rbox_sync_index_rebuild(struct rbox_mailbox *rbox, bool force, std::m extern int rbox_storage_rebuild_in_context(struct rbox_storage *r_storage, bool force, bool firstTry); extern int repair_namespace(struct mail_namespace *ns, bool force, struct rbox_storage *r_storage, std::map> &rados_mails); -extern std::map> load_rados_mail_metadata(bool alt_storage, struct rbox_storage *r_storage, librados::NObjectIterator &iter); +extern std::map> load_rados_mail_metadata(bool alt_storage, struct rbox_storage *r_storage, std::list &mail_list); extern int find_default_mailbox_guid(struct mail_namespace *ns, std::string *mailbox_guid); extern int find_inbox_mailbox_guid(struct mail_namespace *ns, std::string *mailbox_guid); - - -extern std::map> load_rados_mail_metadata(bool alt_storage, struct rbox_storage *r_storage, librados::NObjectIterator &iter); #endif // SRC_STORAGE_RBOX_RBOX_SYNC_REBUILD_H_ diff --git a/src/tests/Makefile.am b/src/tests/Makefile.am index b8399f9f..d90a281c 100644 --- a/src/tests/Makefile.am +++ b/src/tests/Makefile.am @@ -67,6 +67,11 @@ test_storage_mock_rbox_bugs_SOURCES = storage-mock-rbox/test_storage_mock_rbox_b test_storage_mock_rbox_bugs_CPPFLAGS = $(AM_CPPFLAGS) $(LIBDOVECOT_INCLUDE) test_storage_mock_rbox_bugs_LDADD = $(storage_shlibs) $(gtest_shlibs) +TESTS += test_repair_rbox +test_repair_rbox_SOURCES = storage-mock-rbox/test_repair_rbox.cpp storage-mock-rbox/TestCase.cpp storage-mock-rbox/TestCase.h mocks/mock_test.h test-utils/it_utils.cpp test-utils/it_utils.h +test_repair_rbox_CPPFLAGS = $(AM_CPPFLAGS) $(LIBDOVECOT_INCLUDE) +test_repair_rbox_LDADD = $(storage_shlibs) $(gtest_shlibs) + TESTS += test_librmb_utils test_librmb_utils_SOURCES = librmb/test_librmb_utils.cpp @@ -74,6 +79,7 @@ test_librmb_utils_LDADD = $(rmb_shlibs) $(top_builddir)/src/librmb/tools/rmb/ls_ if BUILD_INTEGRATION_TESTS + TESTS += it_test_librmb it_test_librmb_SOURCES = librmb/it_test_librmb.cpp it_test_librmb_LDADD = $(rmb_shlibs) $(top_builddir)/src/librmb/tools/rmb/ls_cmd_parser.o $(top_builddir)/src/librmb/tools/rmb/rmb-commands.o $(top_builddir)/src/librmb/tools/rmb/mailbox_tools.o $(gtest_shlibs) diff --git a/src/tests/mocks/mock_test.h b/src/tests/mocks/mock_test.h index b9a9cfb2..e9e53bb9 100644 --- a/src/tests/mocks/mock_test.h +++ b/src/tests/mocks/mock_test.h @@ -52,6 +52,9 @@ class RadosStorageMock : public RadosStorage { librados::ObjectWriteOperation *op)); MOCK_METHOD1(find_mails, librados::NObjectIterator(const RadosMetadata *attr)); MOCK_METHOD1(open_connection, int(const std::string &poolname)); + + MOCK_METHOD3(find_mails_async, std::set(const RadosMetadata *attr, std::string &pool_name,int num_threads)); + MOCK_METHOD3(open_connection, int(const std::string &poolname, const std::string &clustername, const std::string &rados_username)); MOCK_METHOD0(close_connection, void()); @@ -135,6 +138,8 @@ class RadosClusterMock : public RadosCluster { MOCK_METHOD2(get_config_option, int(const char *option, std::string *value)); MOCK_METHOD0(is_connected, bool()); MOCK_METHOD2(set_config_option, void(const char *option, const char *value)); + MOCK_METHOD1(list_pgs_for_pool, std::vector(std::string &pool_name)); + MOCK_METHOD1(list_pgs_osd_for_pool, std::map> (std::string &pool_name)); }; using librmb::RadosDovecotCephCfg; @@ -155,6 +160,10 @@ class RadosDovecotCephCfgMock : public RadosDovecotCephCfg { MOCK_METHOD0(get_chunk_size,int()); MOCK_METHOD0(get_write_method,int()); + MOCK_METHOD0(get_object_search_method,int()); + + MOCK_METHOD0(get_object_search_threads,int()); + MOCK_METHOD1(update_mail_attributes, void(const char *value)); MOCK_METHOD1(update_updatable_attributes, void(const char *value)); MOCK_METHOD1(update_pool_name_metadata, void(const char *value)); diff --git a/src/tests/storage-mock-rbox/test_repair_rbox.cpp b/src/tests/storage-mock-rbox/test_repair_rbox.cpp new file mode 100644 index 00000000..b435689d --- /dev/null +++ b/src/tests/storage-mock-rbox/test_repair_rbox.cpp @@ -0,0 +1,214 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab +/* + * Copyright (c) 2017-2018 Tallence AG and the authors + * + * This is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License version 2.1, as published by the Free Software + * Foundation. See file COPYING. + */ + +#include "../storage-mock-rbox/TestCase.h" +#include "gtest/gtest.h" +#include "gmock/gmock.h" + +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wshadow" // turn off warnings for Dovecot :-( +#pragma GCC diagnostic ignored "-Wundef" // turn off warnings for Dovecot :-( +#pragma GCC diagnostic ignored "-Wredundant-decls" // turn off warnings for Dovecot :-( +#ifndef __cplusplus +#pragma GCC diagnostic ignored "-Wdeclaration-after-statement" // turn off warnings for Dovecot :-( +#endif + +extern "C" { +#include "lib.h" +#include "mail-user.h" +#include "mail-storage.h" +#include "mail-storage-service.h" +#include "mail-namespace.h" +#include "mailbox-list.h" +#include "ioloop.h" +#include "istream.h" +#include "mail-search-build.h" +#include "ostream.h" +#include "libdict-rados-plugin.h" +} + + +#include "dovecot-ceph-plugin-config.h" +#include "../test-utils/it_utils.h" + +#include "rbox-storage.hpp" +#include "rbox-save.h" + +#include "../mocks/mock_test.h" +#include "rados-util.h" + +using ::testing::_; +using ::testing::AtLeast; +using ::testing::Matcher; +using ::testing::Return; +using ::testing::ReturnRef; +#pragma GCC diagnostic pop + +#if DOVECOT_PREREQ(2, 3) +#define mailbox_get_last_internal_error(box, error_r) mailbox_get_last_internal_error(box, error_r) +#else +#define mailbox_get_last_internal_error(box, error_r) mailbox_get_last_error(box, error_r) +#endif + +#ifndef i_zero +#define i_zero(p) memset(p, 0, sizeof(*(p))) +#endif + +/** + * Parse ls-by-pool output command for pg ids + * + */ +TEST_F(StorageTest, ParseLsByPoolOutput) { + + const std::string lsByPoolOutPut = + "mon command outbl: PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP\n\ + 10.0 568 0 0 0 841115 0 0 1165 active+clean 26h 3641'18906 3656:51938 [3,1,4]p3 [3,1,4]p3 2022-09-12T15:41:31.269234+0000 2022-09-12T15:41:31.269234+0000\n\ + 10.1 550 0 0 0 33248442 0 0 1167 active+clean 25h 3641'22959 3656:56961 [3,2,4]p3 [3,2,4]p3 2022-09-12T17:02:53.075147+0000 2022-09-12T17:02:53.075147+0000\n\ + 10.2 597 0 0 0 1159934 0 0 1323 active+clean 14h 3641'28591 3656:60048 [4,3,2]p4 [4,3,2]p4 2022-09-13T04:19:15.413172+0000 2022-09-11T22:05:48.081770+0000\n\ + 10.3 604 0 0 0 33341614 0 0 1438 active+clean 2h 3641'22933 3657:55462 [1,3,2]p1 [1,3,2]p1 2022-09-13T16:08:56.004228+0000 2022-09-11T09:09:04.723301+0000\n\ + 10.4 562 0 0 0 868945 0 0 1379 active+clean 4h 3641'20673 3656:52381 [1,4,3]p1 [1,4,3]p1 2022-09-13T14:36:42.995722+0000 2022-09-13T14:36:42.995722+0000\n\ + 10.5 603 0 0 0 158821233 0 0 1167 active+clean 18h 3641'117859 3656:149450 [3,4,2]p3 [3,4,2]p3 2022-09-13T00:09:30.875802+0000 2022-09-11T17:26:47.059142+0000\n\ + 10.6 644 0 0 0 933092 0 0 1292 active+clean 43m 3641'22101 3657:54557 [2,1,3]p2 [2,1,3]p2 2022-09-13T17:57:17.071685+0000 2022-09-13T17:57:17.071685+0000\n\ + 10.7 626 0 0 0 95786518 0 0 1193 active+clean 3h 3641'20186 3657:54334 [3,4,1]p3 [3,4,1]p3 2022-09-13T15:22:51.015523+0000 2022-09-12T14:50:10.304250+0000\n\ + 10.8 590 0 0 0 1360275 0 0 1150 active+clean 3h 3641'19029 3657:49696 [3,1,2]p3 [3,1,2]p3 2022-09-13T15:22:28.040315+0000 2022-09-07T06:21:41.088829+0000\n\ + 10.9 575 0 0 0 845156 0 0 1314 active+clean 16h 3641'19322 3656:50842 [2,4,3]p2 [2,4,3]p2 2022-09-13T01:52:08.817572+0000 2022-09-13T01:52:08.817572+0000\n\ + 10.a 567 0 0 0 805954 0 0 1219 active+clean 25h 3641'37553 3656:68282 [3,1,2]p3 [3,1,2]p3 2022-09-12T16:42:57.697431+0000 2022-09-10T10:07:31.790317+0000\n\ + 10.b 582 0 0 0 885900 0 0 1298 active+clean 25h 3641'19280 3656:52838 [2,4,3]p2 [2,4,3]p2 2022-09-12T16:46:16.220323+0000 2022-09-06T17:38:36.977892+0000\n\ + 10.c 549 0 0 0 751071 0 0 1304 active+clean 11h 3641'19066 3656:50539 [2,3,4]p2 [2,3,4]p2 2022-09-13T06:44:32.983447+0000 2022-09-07T18:30:45.260115+0000\n\ + 10.d 650 0 0 0 933800 0 0 1250 active+clean 23h 3641'103478 3656:135352 [2,4,3]p2 [2,4,3]p2 2022-09-12T18:48:06.113121+0000 2022-09-06T20:18:07.792822+0000\n\ + 10.e 566 0 0 0 851780 0 0 1249 active+clean 2h 3532'19569 3657:51205 [2,4,3]p2 [2,4,3]p2 2022-09-13T16:13:32.829316+0000 2022-09-08T14:10:17.974007+0000\n\ + 10.f 597 0 0 0 826336 0 0 1169 active+clean 13h 3641'18988 3656:50480 [3,1,2]p3 [3,1,2]p3 2022-09-13T05:09:49.851413+0000 2022-09-11T19:36:45.701069+0000\n\ + 10.10 562 0 0 0 862878 0 0 1192 active+clean 13h 3641'19245 3656:318916 [3,2,4]p3 [3,2,4]p3 2022-09-13T05:29:59.129259+0000 2022-09-13T05:29:59.129259+0000\n\ + 10.11 547 0 0 0 797004 0 0 1372 active+clean 16h 3641'18587 3656:50156 [1,4,2]p1 [1,4,2]p1 2022-09-13T02:40:21.127749+0000 2022-09-08T08:06:48.604409+0000\n\ + 10.12 530 0 0 0 1052374 0 0 1155 active+clean 28h 3641'20746 3656:51482 [3,1,2]p3 [3,1,2]p3 2022-09-12T14:24:27.828869+0000 2022-09-11T13:33:05.018892+0000\n\ + 10.13 572 0 0 0 847499 0 0 1346 active+clean 15h 3641'28749 3656:60112 [4,3,2]p4 [4,3,2]p4 2022-09-13T03:01:34.151958+0000 2022-09-10T16:11:36.949992+0000\n\ + 10.14 599 0 0 0 1169276 0 0 1279 active+clean 9h 3641'20130 3656:50579 [4,3,1]p4 [4,3,1]p4 2022-09-13T08:48:29.616926+0000 2022-09-13T08:48:29.616926+0000\n\ + 10.15 586 0 0 0 866197 0 0 1279 active+clean 13h 3641'116658 3656:147540 [4,3,2]p4 [4,3,2]p4 2022-09-13T05:36:45.359570+0000 2022-09-08T08:10:42.049640+0000\n\ + 10.16 587 0 0 0 1227527 0 0 1256 active+clean 27h 3641'19293 3656:51660 [2,4,3]p2 [2,4,3]p2 2022-09-12T15:19:09.641685+0000 2022-09-08T21:50:48.032555+0000\n\ + 10.17 566 0 0 0 158772753 0 0 1183 active+clean 7h 3641'34532 3656:65851 [3,1,2]p3 [3,1,2]p3 2022-09-13T10:47:45.614204+0000 2022-09-12T01:21:42.220916+0000\n\ + 10.18 559 0 0 0 63859162 0 0 1354 active+clean 23h 3641'25544 3656:57788 [4,1,2]p4 [4,1,2]p4 2022-09-12T19:35:08.476390+0000 2022-09-11T15:04:20.133155+0000\n\ + 10.19 613 0 0 0 842769 0 0 1166 active+clean 30h 3641'38205 3656:71482 [3,2,4]p3 [3,2,4]p3 2022-09-12T12:21:42.007023+0000 2022-09-06T01:01:59.844062+0000\n\ + 10.1a 587 0 0 0 845231 0 0 1250 active+clean 29h 3641'24127 3656:55136 [2,3,1]p2 [2,3,1]p2 2022-09-12T12:52:01.529860+0000 2022-09-11T07:24:15.003205+0000\n\ + 10.1b 549 0 0 0 76692836 0 0 1316 active+clean 25h 3641'26875 3656:57767 [4,2,3]p4 [4,2,3]p4 2022-09-12T17:06:52.309745+0000 2022-09-08T05:40:28.503848+0000\n\ + 10.1c 604 0 0 0 1114372 0 0 1329 active+clean 62m 3641'18755 3657:50579 [2,3,1]p2 [2,3,1]p2 2022-09-13T17:37:36.331891+0000 2022-09-13T17:37:36.331891+0000\n\ + 10.1d 607 0 0 0 191148409 0 0 1271 active+clean 26h 3641'19526 3656:51525 [2,1,3]p2 [2,1,3]p2 2022-09-12T16:25:02.581960+0000 2022-09-12T16:25:02.581960+0000\n\ + 10.1e 593 0 0 0 848897 0 0 1413 active+clean 20h 3641'19318 3656:51479 [1,3,4]p1 [1,3,4]p1 2022-09-12T22:13:27.455449+0000 2022-09-12T22:13:27.455449+0000\n\ + 10.1f 589 0 0 0 827645 0 0 1288 active+clean 13h 3641'20726 3656:53355 [2,4,1]p2 [2,4,1]p2 2022-09-13T05:09:00.823199+0000 2022-09-07T23:42:54.081441+0000\n"; + + std::vector list = librmb::RadosUtils::extractPgs(lsByPoolOutPut); + + /*for (auto const &token: list) { + std::cout << token << std::endl; + }*/ + + EXPECT_EQ("10.0", list[0]); + EXPECT_EQ("10.1",list[1]); + EXPECT_EQ("10.2",list[2]); + EXPECT_EQ("10.3",list[3]); + EXPECT_EQ("10.4",list[4]); + EXPECT_EQ("10.5",list[5]); + EXPECT_EQ("10.6",list[6]); + EXPECT_EQ("10.7",list[7]); + EXPECT_EQ("10.8",list[8]); + EXPECT_EQ("10.9",list[9]); + EXPECT_EQ("10.a",list[10]); + EXPECT_EQ("10.b",list[11]); + EXPECT_EQ("10.c",list[12]); + EXPECT_EQ("10.d",list[13]); + EXPECT_EQ("10.e",list[14]); + EXPECT_EQ("10.f",list[15]); + EXPECT_EQ("10.10",list[16]); + EXPECT_EQ("10.11",list[17]); + EXPECT_EQ("10.12",list[18]); + EXPECT_EQ("10.13",list[19]); + EXPECT_EQ("10.14",list[20]); + EXPECT_EQ("10.15",list[21]); + EXPECT_EQ("10.16",list[22]); + EXPECT_EQ("10.17",list[23]); + EXPECT_EQ("10.18",list[24]); + EXPECT_EQ("10.19",list[25]); + EXPECT_EQ("10.1a",list[26]); + EXPECT_EQ("10.1b",list[27]); + EXPECT_EQ("10.1c",list[28]); + EXPECT_EQ("10.1d",list[29]); + EXPECT_EQ("10.1e",list[30]); + EXPECT_EQ("10.1f",list[31]); + +} +TEST_F(StorageTest, scanForPg) { + + const std::string header = + "mon command outbl: PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP\n"; + + const std::string row ="10.0 568 0 0 0 841115 0 0 1165 active+clean 26h 3641'18906 3656:51938 [3,1,4]p3 [3,1,4]p3 2022-09-12T15:41:31.269234+0000 2022-09-12T15:41:31.269234+0000\n"; + + std::vector list = librmb::RadosUtils::split(row,' '); + EXPECT_EQ(17,list.size()); + + std::vector list2= librmb::RadosUtils::split(header,' '); + EXPECT_EQ(20,list2.size()); + +} +TEST_F(StorageTest, extractPrimaryOsd) { + + const std::string lsByPoolOutPut = + "mon command outbl: PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP\n\ + 10.0 568 0 0 0 841115 0 0 1165 active+clean 26h 3641'18906 3656:51938 [3,1,4]p3 [3,1,4]p3 2022-09-12T15:41:31.269234+0000 2022-09-12T15:41:31.269234+0000\n\ + 10.1 550 0 0 0 33248442 0 0 1167 active+clean 25h 3641'22959 3656:56961 [3,2,4]p3 [3,2,4]p3 2022-09-12T17:02:53.075147+0000 2022-09-12T17:02:53.075147+0000\n\ + 10.2 597 0 0 0 1159934 0 0 1323 active+clean 14h 3641'28591 3656:60048 [4,3,2]p4 [4,3,2]p4 2022-09-13T04:19:15.413172+0000 2022-09-11T22:05:48.081770+0000\n\ + 10.3 604 0 0 0 33341614 0 0 1438 active+clean 2h 3641'22933 3657:55462 [1,3,2]p1 [1,3,2]p1 2022-09-13T16:08:56.004228+0000 2022-09-11T09:09:04.723301+0000\n\ + 10.4 562 0 0 0 868945 0 0 1379 active+clean 4h 3641'20673 3656:52381 [1,4,3]p1 [1,4,3]p1 2022-09-13T14:36:42.995722+0000 2022-09-13T14:36:42.995722+0000\n\ + 10.5 603 0 0 0 158821233 0 0 1167 active+clean 18h 3641'117859 3656:149450 [3,4,2]p3 [3,4,2]p3 2022-09-13T00:09:30.875802+0000 2022-09-11T17:26:47.059142+0000\n\ + 10.6 644 0 0 0 933092 0 0 1292 active+clean 43m 3641'22101 3657:54557 [2,1,3]p2 [2,1,3]p2 2022-09-13T17:57:17.071685+0000 2022-09-13T17:57:17.071685+0000\n\ + 10.7 626 0 0 0 95786518 0 0 1193 active+clean 3h 3641'20186 3657:54334 [3,4,1]p3 [3,4,1]p3 2022-09-13T15:22:51.015523+0000 2022-09-12T14:50:10.304250+0000\n\ + 10.8 590 0 0 0 1360275 0 0 1150 active+clean 3h 3641'19029 3657:49696 [3,1,2]p3 [3,1,2]p3 2022-09-13T15:22:28.040315+0000 2022-09-07T06:21:41.088829+0000\n\ + 10.9 575 0 0 0 845156 0 0 1314 active+clean 16h 3641'19322 3656:50842 [2,4,3]p2 [2,4,3]p2 2022-09-13T01:52:08.817572+0000 2022-09-13T01:52:08.817572+0000\n\ + 10.a 567 0 0 0 805954 0 0 1219 active+clean 25h 3641'37553 3656:68282 [3,1,2]p3 [3,1,2]p3 2022-09-12T16:42:57.697431+0000 2022-09-10T10:07:31.790317+0000\n\ + 10.b 582 0 0 0 885900 0 0 1298 active+clean 25h 3641'19280 3656:52838 [2,4,3]p2 [2,4,3]p2 2022-09-12T16:46:16.220323+0000 2022-09-06T17:38:36.977892+0000\n\ + 10.c 549 0 0 0 751071 0 0 1304 active+clean 11h 3641'19066 3656:50539 [2,3,4]p2 [2,3,4]p2 2022-09-13T06:44:32.983447+0000 2022-09-07T18:30:45.260115+0000\n\ + 10.d 650 0 0 0 933800 0 0 1250 active+clean 23h 3641'103478 3656:135352 [2,4,3]p2 [2,4,3]p2 2022-09-12T18:48:06.113121+0000 2022-09-06T20:18:07.792822+0000\n\ + 10.e 566 0 0 0 851780 0 0 1249 active+clean 2h 3532'19569 3657:51205 [2,4,3]p2 [2,4,3]p2 2022-09-13T16:13:32.829316+0000 2022-09-08T14:10:17.974007+0000\n\ + 10.f 597 0 0 0 826336 0 0 1169 active+clean 13h 3641'18988 3656:50480 [3,1,2]p3 [3,1,2]p3 2022-09-13T05:09:49.851413+0000 2022-09-11T19:36:45.701069+0000\n\ + 10.10 562 0 0 0 862878 0 0 1192 active+clean 13h 3641'19245 3656:318916 [3,2,4]p3 [3,2,4]p3 2022-09-13T05:29:59.129259+0000 2022-09-13T05:29:59.129259+0000\n\ + 10.11 547 0 0 0 797004 0 0 1372 active+clean 16h 3641'18587 3656:50156 [1,4,2]p1 [1,4,2]p1 2022-09-13T02:40:21.127749+0000 2022-09-08T08:06:48.604409+0000\n\ + 10.12 530 0 0 0 1052374 0 0 1155 active+clean 28h 3641'20746 3656:51482 [3,1,2]p3 [3,1,2]p3 2022-09-12T14:24:27.828869+0000 2022-09-11T13:33:05.018892+0000\n\ + 10.13 572 0 0 0 847499 0 0 1346 active+clean 15h 3641'28749 3656:60112 [4,3,2]p4 [4,3,2]p4 2022-09-13T03:01:34.151958+0000 2022-09-10T16:11:36.949992+0000\n\ + 10.14 599 0 0 0 1169276 0 0 1279 active+clean 9h 3641'20130 3656:50579 [4,3,1]p4 [4,3,1]p4 2022-09-13T08:48:29.616926+0000 2022-09-13T08:48:29.616926+0000\n\ + 10.15 586 0 0 0 866197 0 0 1279 active+clean 13h 3641'116658 3656:147540 [4,3,2]p4 [4,3,2]p4 2022-09-13T05:36:45.359570+0000 2022-09-08T08:10:42.049640+0000\n\ + 10.16 587 0 0 0 1227527 0 0 1256 active+clean 27h 3641'19293 3656:51660 [2,4,3]p2 [2,4,3]p2 2022-09-12T15:19:09.641685+0000 2022-09-08T21:50:48.032555+0000\n\ + 10.17 566 0 0 0 158772753 0 0 1183 active+clean 7h 3641'34532 3656:65851 [3,1,2]p3 [3,1,2]p3 2022-09-13T10:47:45.614204+0000 2022-09-12T01:21:42.220916+0000\n\ + 10.18 559 0 0 0 63859162 0 0 1354 active+clean 23h 3641'25544 3656:57788 [4,1,2]p4 [4,1,2]p4 2022-09-12T19:35:08.476390+0000 2022-09-11T15:04:20.133155+0000\n\ + 10.19 613 0 0 0 842769 0 0 1166 active+clean 30h 3641'38205 3656:71482 [3,2,4]p3 [3,2,4]p3 2022-09-12T12:21:42.007023+0000 2022-09-06T01:01:59.844062+0000\n\ + 10.1a 587 0 0 0 845231 0 0 1250 active+clean 29h 3641'24127 3656:55136 [2,3,1]p2 [2,3,1]p2 2022-09-12T12:52:01.529860+0000 2022-09-11T07:24:15.003205+0000\n\ + 10.1b 549 0 0 0 76692836 0 0 1316 active+clean 25h 3641'26875 3656:57767 [4,2,3]p4 [4,2,3]p4 2022-09-12T17:06:52.309745+0000 2022-09-08T05:40:28.503848+0000\n\ + 10.1c 604 0 0 0 1114372 0 0 1329 active+clean 62m 3641'18755 3657:50579 [2,3,1]p2 [2,3,1]p2 2022-09-13T17:37:36.331891+0000 2022-09-13T17:37:36.331891+0000\n\ + 10.1d 607 0 0 0 191148409 0 0 1271 active+clean 26h 3641'19526 3656:51525 [2,1,3]p2 [2,1,3]p2 2022-09-12T16:25:02.581960+0000 2022-09-12T16:25:02.581960+0000\n\ + 10.1e 593 0 0 0 848897 0 0 1413 active+clean 20h 3641'19318 3656:51479 [1,3,4]p1 [1,3,4]p1 2022-09-12T22:13:27.455449+0000 2022-09-12T22:13:27.455449+0000\n\ + 10.1f 589 0 0 0 827645 0 0 1288 active+clean 13h 3641'20726 3656:53355 [2,4,1]p2 [2,4,1]p2 2022-09-13T05:09:00.823199+0000 2022-09-07T23:42:54.081441+0000\n"; + + std::map> list = librmb::RadosUtils::extractPgAndPrimaryOsd(lsByPoolOutPut); + + for (const auto& x : list) + { + std::cout << "first: " << x.first << ", second: " << x.second.size() << std::endl; + } + + EXPECT_EQ(4,list["1"].size()); + EXPECT_EQ(11,list["2"].size()); + EXPECT_EQ(11,list["3"].size()); + EXPECT_EQ(6,list["4"].size()); + +} + +int main(int argc, char **argv) { + ::testing::InitGoogleMock(&argc, argv); + return RUN_ALL_TESTS(); +}