Feature/342 object search (#343)

* Develop (#331) * #328: fix copy move (#330) * Develop (#334) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * Bugfix/283 virtual mailbox fetch metadata (#284) * Develop (#282) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * #283: fetch metadata for mails in virtual mailbox * #283: fix metadata date.saved. date.received virtual mailbox * #283: version * Feature/286 use guid from UUID string (#287) * #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing * version 0.0.25 preparations * #286: build issue * merge * version * #286: prefere cached mail guid. * #286: determine if uuid has - hyphon, if true preseve it. * fix non void return * Feature/289 GitHub actions (#290) * #289: build * #289: build plugin. * #289: submodules * submodules * upgrade git * clean outdated repos * Bugfix/UUID record format (#293) * bugfix initialisaction rados_mail->deprecated_uid * rados_mail creation and default value check * #295: update index after full object has been moved (metadata) (#296) * Bugfix/298 295 bugfixes (#300) * #289: fix force-resync (always use INBOX) #295: rbox_mail_set_expunge (play save) * #298: move ceph objects the standard way. * clean up log. * #298: log warning if client connection times out. * Feature/302 retry expunge connection timeout (#303) * #302: retry in case of ceph connection timeout * rpm * #302: max_retry = 10 * max 10 retry, in case object not available, fail with error * #304: use last know uid to restore email flags (#305) * #306: re-assign unreferenced mail objects to inbox (#309) * #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311) * version: 0.0.31 * #313: fix imap append crash (#314) * version 0.0.32 * Bugfix/316 mailbox save (#317) * #316: imap append (remove index entry twice in case of error) * #316: changed storage for big attachments. (sync write) * #316: operate api * #316: fix operate call * #316: release preparations * #316: fixed some legacy unit tests * 316: review results * #316: fix * Bugfix/test output stream check (#323) * init output stream and free mailbuffer * build * extracted write chunks * mailsize -1; * fix * debug messages * init stream any case * added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes * missing mock class * version * disabled (temporarily storage tests) * disabled storage tests( api change) * Feature/alternative save methods (#326) * #322: alternative save methods * version * Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321) * #319: automatically add lost objects to inbox (if no mailbox guid exist) * #319: version 0.0.34 * catch error in case mailbox does not have obox header: e.v. Virtual Mailbox * check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin) * version * #328: fix copy move (#330) * Bugfix/332 quota and move (#333) * 291: failed save mock test * 291: fixed most of the tests * 332 fix notify messages number and type * Develop (#336) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * Bugfix/283 virtual mailbox fetch metadata (#284) * Develop (#282) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * #283: fetch metadata for mails in virtual mailbox * #283: fix metadata date.saved. date.received virtual mailbox * #283: version * Feature/286 use guid from UUID string (#287) * #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing * version 0.0.25 preparations * #286: build issue * merge * version * #286: prefere cached mail guid. * #286: determine if uuid has - hyphon, if true preseve it. * fix non void return * Feature/289 GitHub actions (#290) * #289: build * #289: build plugin. * #289: submodules * submodules * upgrade git * clean outdated repos * Bugfix/UUID record format (#293) * bugfix initialisaction rados_mail->deprecated_uid * rados_mail creation and default value check * #295: update index after full object has been moved (metadata) (#296) * Bugfix/298 295 bugfixes (#300) * #289: fix force-resync (always use INBOX) #295: rbox_mail_set_expunge (play save) * #298: move ceph objects the standard way. * clean up log. * #298: log warning if client connection times out. * Feature/302 retry expunge connection timeout (#303) * #302: retry in case of ceph connection timeout * rpm * #302: max_retry = 10 * max 10 retry, in case object not available, fail with error * #304: use last know uid to restore email flags (#305) * #306: re-assign unreferenced mail objects to inbox (#309) * #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311) * version: 0.0.31 * #313: fix imap append crash (#314) * version 0.0.32 * Bugfix/316 mailbox save (#317) * #316: imap append (remove index entry twice in case of error) * #316: changed storage for big attachments. (sync write) * #316: operate api * #316: fix operate call * #316: release preparations * #316: fixed some legacy unit tests * 316: review results * #316: fix * Bugfix/test output stream check (#323) * init output stream and free mailbuffer * build * extracted write chunks * mailsize -1; * fix * debug messages * init stream any case * added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes * missing mock class * version * disabled (temporarily storage tests) * disabled storage tests( api change) * Feature/alternative save methods (#326) * #322: alternative save methods * version * Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321) * #319: automatically add lost objects to inbox (if no mailbox guid exist) * #319: version 0.0.34 * catch error in case mailbox does not have obox header: e.v. Virtual Mailbox * check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin) * version * #328: fix copy move (#330) * Bugfix/332 quota and move (#333) * 291: failed save mock test * 291: fixed most of the tests * 332 fix notify messages number and type * #332: fix append additional error mssage in case connection can't be opend (#335) * Develop (#341) * Loglevel Debug is sufficient as it gets retried (#337) * Bugfix/339 rados config timeout (#340) * 339 retry 10 times with random wait. retry read for simple read e.g. rados_config. fail with assert in case we can't find rados_config * #342: simple multithreading object search * #342: version 0.0.40 Co-authored-by: Ewald Dieterich <[email protected]>
ceph-dovecot · Sep 22, 2022 · dde2fb3 · dde2fb3
1 parent 690127d
commit dde2fb3
Show file tree

Hide file tree

Showing 21 changed files with 517 additions and 23 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,13 @@
 # Change Log
 
+## [0.0.40](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.39) (2022-09-22)
+- #342 multithreading object search for doveadm force-resync (feature toggle) 
+       new config params: 
+       # search method default = 0 | 1 multithreading
+       rbox_object_search_method=1 
+       # number of threads to use in case of search_method=1
+       rbox_object_search_threads=4
+
 ## [0.0.39](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.39) (2022-08-25)
 - #339 fail with assert if rados_config cannot be found due to network/connection issue 
        retry ceph read operations / read / xattr with timeout 

diff --git a/configure.ac b/configure.ac
@@ -9,7 +9,7 @@
 AC_PREREQ([2.59])
 
 
-AC_INIT([dovecot-ceph-plugin], [0.0.39], [https://github.com/ceph-dovecot/dovecot-ceph-plugin/issues/new], ,[https://github.com/ceph-dovecot/dovecot-ceph-plugin])
+AC_INIT([dovecot-ceph-plugin], [0.0.40], [https://github.com/ceph-dovecot/dovecot-ceph-plugin/issues/new], ,[https://github.com/ceph-dovecot/dovecot-ceph-plugin])
 
 
 AC_CONFIG_AUX_DIR([.])

diff --git a/rpm/dovecot-ceph-plugin.spec b/rpm/dovecot-ceph-plugin.spec
@@ -15,7 +15,7 @@ Name:		dovecot-ceph-plugin
 Summary:	Dovecot Ceph RADOS plugins
 
 
-Version:	0.0.39
+Version:	0.0.40
 
 Release:	0%{?dist}
 URL:		https://github.com/ceph-dovecot/dovecot-ceph-plugin

diff --git a/src/librmb/rados-ceph-config.h b/src/librmb/rados-ceph-config.h
@@ -58,7 +58,7 @@ class RadosCephConfig {
   void set_update_attributes(const std::string &update_attributes_) {
     config.set_update_attributes(update_attributes_);
   }
-
+  
   void update_mail_attribute(const char *value) { config.update_mail_attribute(value); }
   void update_updateable_attribute(const char *value) { config.update_updateable_attribute(value); }
 

diff --git a/src/librmb/rados-cluster-impl.cpp b/src/librmb/rados-cluster-impl.cpp
@@ -17,7 +17,7 @@
 
 #include "rados-dictionary-impl.h"
 #include "rados-storage-impl.h"
-
+#include "rados-util.h"
 using std::list;
 using std::pair;
 using std::string;
@@ -72,6 +72,57 @@ int RadosClusterImpl::init(const std::string &clustername, const std::string &ra
   return ret;
 }
 
+
+std::vector<std::string> RadosClusterImpl::list_pgs_for_pool(std::string &pool_name) {
+    std::cout << " ola "  << RadosClusterImpl::cluster << std::endl;
+
+    if(is_connected()){
+      std::cout << " is connected YES" << std::endl;
+    }else{
+      std::cout << " is connected NO" << std::endl;
+      connect();
+    }
+
+    const string pool = "mail_storage";
+    const string cmd =
+    "{"
+    "\"prefix\": \"pg ls-by-pool\", "
+    "\"poolstr\": \"" + pool + "\""
+    "}";      
+
+    std::cout << "cmd: " << cmd << std::endl;
+
+    librados::bufferlist inbl;
+    librados::bufferlist outbl;
+    int res = RadosClusterImpl::cluster->mon_command(cmd, inbl, &outbl, nullptr);
+    std::cout << "inbl command " << inbl  <<std::endl;
+    std::cout << "outbl command " << outbl.c_str()  <<std::endl;
+
+    std::vector<std::string> list = RadosUtils::extractPgs(std::string(outbl.c_str()));
+
+    for (auto const &token: list) {
+          std::cout << token << std::endl;        
+    }
+    return list;
+}
+
+std::map<std::string, std::vector<std::string>> RadosClusterImpl::list_pgs_osd_for_pool(std::string &pool_name) {    
+
+    if(!is_connected()){      
+      connect();
+    }
+
+    const string cmd =
+    "{"
+    "\"prefix\": \"pg ls-by-pool\", "
+    "\"poolstr\": \"" + pool_name + "\""
+    "}";      
+
+    librados::bufferlist inbl;
+    librados::bufferlist outbl;
+    RadosClusterImpl::cluster->mon_command(cmd, inbl, &outbl, nullptr);
+    return RadosUtils::extractPgAndPrimaryOsd(std::string(outbl.c_str()));
+}
 int RadosClusterImpl::initialize() {
   int ret = 0;
 

diff --git a/src/librmb/rados-cluster-impl.h b/src/librmb/rados-cluster-impl.h
@@ -40,6 +40,9 @@ class RadosClusterImpl : public RadosCluster {
   librados::Rados &get_cluster() { return *cluster; }
   void set_config_option(const char *option, const char *value);
 
+  std::vector<std::string> list_pgs_for_pool(std::string &pool_name) override;
+  std::map<std::string, std::vector<std::string>> list_pgs_osd_for_pool(std::string &pool_name) override;
+
  private:
   int initialize();
 

diff --git a/src/librmb/rados-cluster.h b/src/librmb/rados-cluster.h
@@ -76,6 +76,13 @@ class RadosCluster {
    * @return true if connected
    */
   virtual bool is_connected() = 0;
+
+  /*! get placement groups for mailbox storage pool 
+  */
+  virtual std::vector<std::string> list_pgs_for_pool(std::string &pool_name) = 0;
+  virtual std::map<std::string, std::vector<std::string>> list_pgs_osd_for_pool(std::string &pool_name) = 0;
+
+
 };
 
 }  // namespace librmb

diff --git a/src/librmb/rados-dovecot-ceph-cfg-impl.h b/src/librmb/rados-dovecot-ceph-cfg-impl.h
@@ -58,6 +58,8 @@ class RadosDovecotCephCfgImpl : public RadosDovecotCephCfg {
       rados_cfg.set_cfg_object_name(dovecot_cfg.get_rbox_cfg_object_name());
     }
   }
+  int get_object_search_method()  override { return std::stoi(dovecot_cfg.get_object_search_method()); }
+  int get_object_search_threads() override { return std::stoi(dovecot_cfg.get_object_search_threads()); }
 
   void set_rbox_cfg_object_name(const std::string &value) override { dovecot_cfg.set_rbox_cfg_object_name(value); }
 

diff --git a/src/librmb/rados-dovecot-ceph-cfg.h b/src/librmb/rados-dovecot-ceph-cfg.h
@@ -44,6 +44,9 @@ class RadosDovecotCephCfg {
   virtual int get_chunk_size() = 0;
   virtual int get_write_method() = 0;
 
+  virtual int get_object_search_method()  = 0;
+  virtual int get_object_search_threads() = 0;
+
   virtual const std::string &get_pool_name_metadata_key() = 0;
   virtual const std::string &get_update_attributes_key() = 0;
   virtual const std::string &get_mail_attributes_key() = 0;

diff --git a/src/librmb/rados-dovecot-config.cpp b/src/librmb/rados-dovecot-config.cpp
@@ -29,7 +29,9 @@ RadosConfig::RadosConfig()
       rbox_ceph_aio_wait_for_safe_and_cb("rbox_ceph_aio_wait_for_safe_and_cb"),
       rbox_ceph_write_chunks("rbox_ceph_write_chunks"),
       rbox_chunk_size("rbox_chunk_size"),
-      rbox_write_method("rbox_write_method") {
+      rbox_write_method("rbox_write_method"),
+      rbox_object_search_method("rbox_object_search_method"),
+      rbox_object_search_threads("rbox_object_search_threads") {
 
   config[pool_name] = "mail_storage";
 
@@ -43,6 +45,9 @@ RadosConfig::RadosConfig()
   config[rbox_ceph_write_chunks] = "false";
   config[rbox_chunk_size] = "10240";
   config[rbox_write_method] = "0";
+  config[rbox_object_search_method] = "0";
+  config[rbox_object_search_threads] = "4";
+
   is_valid = false;
 }
 
@@ -79,8 +84,10 @@ std::string RadosConfig::to_string() {
   ss << "  " << rbox_ceph_aio_wait_for_safe_and_cb << "=" << config[rbox_ceph_aio_wait_for_safe_and_cb] << std::endl;
   ss << "  " << rbox_ceph_write_chunks << "=" << config[rbox_ceph_write_chunks] << std::endl;
   ss << "  " << rbox_write_method << "=" << config[rbox_write_method] << std::endl;
-  ss << "  " << rbox_chunk_size << "=" << config[rbox_chunk_size]
-     << std::endl;
+  ss << "  " << rbox_chunk_size << "=" << config[rbox_chunk_size] << std::endl;
+  ss << "  " << rbox_object_search_method << "=" << config[rbox_object_search_method] << std::endl;
+  ss << "  " << rbox_object_search_threads << "=" << config[rbox_object_search_threads] << std::endl;
+
   return ss.str();
 }
 

diff --git a/src/librmb/rados-dovecot-config.h b/src/librmb/rados-dovecot-config.h
@@ -49,6 +49,10 @@ class RadosConfig {
 
   const std::string &get_rbox_cluster_name() { return config[rbox_cluster_name]; }
   const std::string &get_rados_username() { return config[rados_username]; }
+
+  const std::string &get_object_search_method()  { return config[rbox_object_search_method]; }
+  const std::string &get_object_search_threads() { return config[rbox_object_search_threads]; }
+
   void update_metadata(const std::string &key, const char *value_);
   bool is_ceph_posix_bugfix_enabled() {
     return config[bugfix_cephfs_posix_hardlinks].compare("true") == 0 ? true : false;
@@ -87,6 +91,8 @@ class RadosConfig {
   std::string rbox_ceph_write_chunks;
   std::string rbox_chunk_size;
   std::string rbox_write_method;
+  std::string rbox_object_search_method;
+  std::string rbox_object_search_threads;
   bool is_valid;
 };
 

diff --git a/src/librmb/rados-storage-impl.cpp b/src/librmb/rados-storage-impl.cpp
@@ -16,8 +16,13 @@
 #include <set>
 #include <string>
 #include <utility>
+#include <thread>
+#include <mutex>
+
+#include "rados-util.h"
 
 #include <rados/librados.hpp>
+
 #include "encoding.h"
 #include "limits.h"
 
@@ -173,7 +178,6 @@ librados::NObjectIterator RadosStorageImpl::find_mails(const RadosMetadata *attr
   }
 
   if (attr != nullptr) {
-   // int hashpos = get_io_ctx().get_object_hash_position("t1_u");
     std::string filter_name = PLAIN_FILTER_NAME;
     ceph::bufferlist filter_bl;
 
@@ -186,7 +190,57 @@ librados::NObjectIterator RadosStorageImpl::find_mails(const RadosMetadata *attr
     return get_io_ctx().nobjects_begin();
   }
 }
-
+/**
+ * POC Implementation: 
+ * 
+ * see in prod how it behaves. 
+ * 
+ **/
+std::set<std::string> RadosStorageImpl::find_mails_async(const RadosMetadata *attr, 
+                                                         std::string &pool_name,
+                                                         int num_threads){
+
+    std::set<std::string> oid_list;
+    std::mutex oid_list_mutex;
+
+    // Define a Lambda Expression
+    auto f = [](const std::vector<std::string> &list, std::mutex &oid_mutex, std::set<std::string> &oids, librados::IoCtx *io_ctx) {
+
+        std::lock_guard<std::mutex> guard(oid_mutex);
+        for (auto const &pg: list) {
+          uint64_t ppool;
+          uint32_t pseed;
+          int r = sscanf(pg.c_str(), "%llu.%x", (long long unsigned *)&ppool, &pseed);
+
+          librados::NObjectIterator iter= io_ctx->nobjects_begin(pseed);
+
+          while (iter != librados::NObjectIterator::__EndObjectIterator) {
+            std::string oid = iter->get_oid();
+            oids.insert(oid);  
+            iter++;
+          }          
+        }           
+    };
+
+    //std::string pool_mame = "mail_storage";
+    std::map<std::string, std::vector<std::string>> osd_pg_map = cluster->list_pgs_osd_for_pool(pool_name);
+    std::vector<std::thread> threads;
+
+    for (const auto& x : osd_pg_map)
+    {
+      if(threads.size() == num_threads){
+        for (auto const &thread: threads) {
+          thread.join();
+        }   
+        threads.clear();      
+      }
+      threads.push_back(std::thread(f, std::ref(x.second),std::ref(oid_list_mutex),std::ref(oid_list), &get_io_ctx()));
+    }
+    for (auto const &thread: threads) {
+        thread.join();
+    }   
+    return oid_list;
+}
 librados::IoCtx &RadosStorageImpl::get_io_ctx() { return io_ctx; }
 
 int RadosStorageImpl::open_connection(const std::string &poolname, const std::string &clustername,

diff --git a/src/librmb/rados-storage-impl.h b/src/librmb/rados-storage-impl.h
@@ -18,6 +18,8 @@
 #include <string>
 #include <cstdint>
 #include <list>
+#include <set>
+
 #include <algorithm>
 #include <rados/librados.hpp>
 
@@ -50,6 +52,9 @@ class RadosStorageImpl : public RadosStorage {
   int aio_operate(librados::IoCtx *io_ctx_, const std::string &oid, librados::AioCompletion *c,
                   librados::ObjectWriteOperation *op) override;
   librados::NObjectIterator find_mails(const RadosMetadata *attr) override;
+
+  std::set<std::string> find_mails_async(const RadosMetadata *attr, std::string &pool_name, int num_threads) override;
+
   int open_connection(const std::string &poolname) override;
   int open_connection(const std::string &poolname, const std::string &clustername,
                       const std::string &rados_username) override;
@@ -72,6 +77,7 @@ class RadosStorageImpl : public RadosStorage {
 
   void free_rados_mail(librmb::RadosMail *mail) override;
 
+
  private:
   int create_connection(const std::string &poolname);
 

diff --git a/src/librmb/rados-storage.h b/src/librmb/rados-storage.h
@@ -104,6 +104,11 @@ class RadosStorage {
    *
    * @return object iterator or librados::NObjectIterator::__EndObjectIterator */
   virtual librados::NObjectIterator find_mails(const RadosMetadata *attr) = 0;
+
+
+  virtual std::set<std::string> find_mails_async(const RadosMetadata *attr, std::string &pool_name, int num_threads) = 0;
+
+
   /*! open the rados connections with default cluster and username
    * @param[in] poolname the poolname to connect to, in case this one does not exists, it will be created.
    * */
@@ -199,6 +204,7 @@ class RadosStorage {
    * */
   virtual void free_rados_mail(librmb::RadosMail *mail) = 0;
 
+
 };
 
 }  // namespace librmb