Develop (#348)

* Loglevel Debug is sufficient as it gets retried (#337) * Bugfix/339 rados config timeout (#340) * 339 retry 10 times with random wait. retry read for simple read e.g. rados_config. fail with assert in case we can't find rados_config * Feature/342 object search (#343) * Develop (#331) * #328: fix copy move (#330) * Develop (#334) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * Bugfix/283 virtual mailbox fetch metadata (#284) * Develop (#282) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * #283: fetch metadata for mails in virtual mailbox * #283: fix metadata date.saved. date.received virtual mailbox * #283: version * Feature/286 use guid from UUID string (#287) * #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing * version 0.0.25 preparations * #286: build issue * merge * version * #286: prefere cached mail guid. * #286: determine if uuid has - hyphon, if true preseve it. * fix non void return * Feature/289 GitHub actions (#290) * #289: build * #289: build plugin. * #289: submodules * submodules * upgrade git * clean outdated repos * Bugfix/UUID record format (#293) * bugfix initialisaction rados_mail->deprecated_uid * rados_mail creation and default value check * #295: update index after full object has been moved (metadata) (#296) * Bugfix/298 295 bugfixes (#300) * #289: fix force-resync (always use INBOX) #295: rbox_mail_set_expunge (play save) * #298: move ceph objects the standard way. * clean up log. * #298: log warning if client connection times out. * Feature/302 retry expunge connection timeout (#303) * #302: retry in case of ceph connection timeout * rpm * #302: max_retry = 10 * max 10 retry, in case object not available, fail with error * #304: use last know uid to restore email flags (#305) * #306: re-assign unreferenced mail objects to inbox (#309) * #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311) * version: 0.0.31 * #313: fix imap append crash (#314) * version 0.0.32 * Bugfix/316 mailbox save (#317) * #316: imap append (remove index entry twice in case of error) * #316: changed storage for big attachments. (sync write) * #316: operate api * #316: fix operate call * #316: release preparations * #316: fixed some legacy unit tests * 316: review results * #316: fix * Bugfix/test output stream check (#323) * init output stream and free mailbuffer * build * extracted write chunks * mailsize -1; * fix * debug messages * init stream any case * added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes * missing mock class * version * disabled (temporarily storage tests) * disabled storage tests( api change) * Feature/alternative save methods (#326) * #322: alternative save methods * version * Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321) * #319: automatically add lost objects to inbox (if no mailbox guid exist) * #319: version 0.0.34 * catch error in case mailbox does not have obox header: e.v. Virtual Mailbox * check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin) * version * #328: fix copy move (#330) * Bugfix/332 quota and move (#333) * 291: failed save mock test * 291: fixed most of the tests * 332 fix notify messages number and type * Develop (#336) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * Bugfix/283 virtual mailbox fetch metadata (#284) * Develop (#282) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * #283: fetch metadata for mails in virtual mailbox * #283: fix metadata date.saved. date.received virtual mailbox * #283: version * Feature/286 use guid from UUID string (#287) * #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing * version 0.0.25 preparations * #286: build issue * merge * version * #286: prefere cached mail guid. * #286: determine if uuid has - hyphon, if true preseve it. * fix non void return * Feature/289 GitHub actions (#290) * #289: build * #289: build plugin. * #289: submodules * submodules * upgrade git * clean outdated repos * Bugfix/UUID record format (#293) * bugfix initialisaction rados_mail->deprecated_uid * rados_mail creation and default value check * #295: update index after full object has been moved (metadata) (#296) * Bugfix/298 295 bugfixes (#300) * #289: fix force-resync (always use INBOX) #295: rbox_mail_set_expunge (play save) * #298: move ceph objects the standard way. * clean up log. * #298: log warning if client connection times out. * Feature/302 retry expunge connection timeout (#303) * #302: retry in case of ceph connection timeout * rpm * #302: max_retry = 10 * max 10 retry, in case object not available, fail with error * #304: use last know uid to restore email flags (#305) * #306: re-assign unreferenced mail objects to inbox (#309) * #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311) * version: 0.0.31 * #313: fix imap append crash (#314) * version 0.0.32 * Bugfix/316 mailbox save (#317) * #316: imap append (remove index entry twice in case of error) * #316: changed storage for big attachments. (sync write) * #316: operate api * #316: fix operate call * #316: release preparations * #316: fixed some legacy unit tests * 316: review results * #316: fix * Bugfix/test output stream check (#323) * init output stream and free mailbuffer * build * extracted write chunks * mailsize -1; * fix * debug messages * init stream any case * added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes * missing mock class * version * disabled (temporarily storage tests) * disabled storage tests( api change) * Feature/alternative save methods (#326) * #322: alternative save methods * version * Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321) * #319: automatically add lost objects to inbox (if no mailbox guid exist) * #319: version 0.0.34 * catch error in case mailbox does not have obox header: e.v. Virtual Mailbox * check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin) * version * #328: fix copy move (#330) * Bugfix/332 quota and move (#333) * 291: failed save mock test * 291: fixed most of the tests * 332 fix notify messages number and type * #332: fix append additional error mssage in case connection can't be opend (#335) * Develop (#341) * Loglevel Debug is sufficient as it gets retried (#337) * Bugfix/339 rados config timeout (#340) * 339 retry 10 times with random wait. retry read for simple read e.g. rados_config. fail with assert in case we can't find rados_config * #342: simple multithreading object search * #342: version 0.0.40 Co-authored-by: Ewald Dieterich <[email protected]> * Bugfix/342 logging multithreading (#344) * #342 fix threading and additional logs * version * Bugfix/rbox copy context (#347) * 346; fix rbox-copy seg fault always init rbox_mail * #346 fix segmentation fault rbox_copy * #364 and code cleanup * Feature/349 ceph object index (#350) * #349: initial storage api * #349 doveadm rmb create ceph index force resync strategy * allow refresh of ceph object index via -r param * #349 save ceph index objects to separate pool * #349 poc impl ceph index done * #349 new config setting rbox_index_pool_name * #349 force reload of all objects, config setting defines if append of oid to ceph index happens * #349 add , to every entry or entry list * #349 add , to every entry or entry list, and test * repair invalid object not debug level log * #349 changed the remove function, initial index creation: append after each mailbox * #349 fix unit tests * #349 version Co-authored-by: Ewald Dieterich <[email protected]>
ceph-dovecot · Nov 21, 2022 · 7f400f8 · 7f400f8
1 parent 28b6532
commit 7f400f8
Show file tree

Hide file tree

Showing 32 changed files with 905 additions and 462 deletions.
diff --git a/.gitignore b/.gitignore
@@ -5,6 +5,7 @@ dovecot-rados-plugin-config.h
 .project
 .cproject
 .autotools
+**/cscope*
 
 # http://www.gnu.org/software/automake
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,4 +1,9 @@
 # Change Log
+## [0.0.44](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.44) (2022-11-21)
+- #349 additional recovery method (ceph index object)
+
+## [0.0.43](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.43) (2022-10-27)
+- #346 segmentation fault (rbox_copy) if rbox_mail is null
 
 ## [0.0.42](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.42) (2022-10-18)
 - #346 segmentation fault (rbox_copy) if rbox_mail is null

diff --git a/configure.ac b/configure.ac
@@ -9,7 +9,7 @@
 AC_PREREQ([2.59])
 
 
-AC_INIT([dovecot-ceph-plugin], [0.0.42], [https://github.com/ceph-dovecot/dovecot-ceph-plugin/issues/new], ,[https://github.com/ceph-dovecot/dovecot-ceph-plugin])
+AC_INIT([dovecot-ceph-plugin], [0.0.44], [https://github.com/ceph-dovecot/dovecot-ceph-plugin/issues/new], ,[https://github.com/ceph-dovecot/dovecot-ceph-plugin])
 
 AC_CONFIG_AUX_DIR([.])
 AC_CONFIG_SRCDIR([src])

diff --git a/rpm/dovecot-ceph-plugin.spec b/rpm/dovecot-ceph-plugin.spec
@@ -14,9 +14,7 @@
 Name:		dovecot-ceph-plugin
 Summary:	Dovecot Ceph RADOS plugins
 
-
-Version:	0.0.42
-
+Version:	0.0.44
 
 Release:	0%{?dist}
 URL:		https://github.com/ceph-dovecot/dovecot-ceph-plugin

diff --git a/src/librmb/rados-cluster-impl.cpp b/src/librmb/rados-cluster-impl.cpp
@@ -82,22 +82,17 @@ std::vector<std::string> RadosClusterImpl::list_pgs_for_pool(std::string &pool_n
       std::cout << " is connected NO" << std::endl;
       connect();
     }
-
-    const string pool = "mail_storage";
+
     const string cmd =
     "{"
     "\"prefix\": \"pg ls-by-pool\", "
-    "\"poolstr\": \"" + pool + "\""
+    "\"poolstr\": \"" + pool_name + "\""
     "}";      
-
-    std::cout << "cmd: " << cmd << std::endl;
-
+
     librados::bufferlist inbl;
     librados::bufferlist outbl;
-    int res = RadosClusterImpl::cluster->mon_command(cmd, inbl, &outbl, nullptr);
-    std::cout << "inbl command " << inbl  <<std::endl;
-    std::cout << "outbl command " << outbl.c_str()  <<std::endl;
-
+    RadosClusterImpl::cluster->mon_command(cmd, inbl, &outbl, nullptr);
+
     std::vector<std::string> list = RadosUtils::extractPgs(std::string(outbl.c_str()));
 
     for (auto const &token: list) {
@@ -227,6 +222,18 @@ int RadosClusterImpl::io_ctx_create(const string &pool, librados::IoCtx *io_ctx)
   }
   return ret;
 }
+int RadosClusterImpl::recovery_index_io_ctx(const std::string &pool, 
+  librados::IoCtx *io_ctx) {
+    if(!is_connected()) {
+      return -1;
+    }
+    // pool exists? else create
+    int ret = pool_create(pool);
+    if (ret == 0) {
+      ret = RadosClusterImpl::cluster->ioctx_create(pool.c_str(), *io_ctx);
+    }
+    return ret;      
+}
 
 int RadosClusterImpl::get_config_option(const char *option, string *value) {
   return RadosClusterImpl::cluster->conf_get(option, *value);

diff --git a/src/librmb/rados-cluster-impl.h b/src/librmb/rados-cluster-impl.h
@@ -33,6 +33,8 @@ class RadosClusterImpl : public RadosCluster {
 
   int pool_create(const std::string &pool) override;
   int io_ctx_create(const std::string &pool, librados::IoCtx *io_ctx) override;
+  int recovery_index_io_ctx(const std::string &pool, librados::IoCtx *io_ctx) override;
+
   int get_config_option(const char *option, std::string *value) override;
   int dictionary_create(const std::string &pool, const std::string &username, const std::string &oid,
                         RadosDictionary **dictionary);

diff --git a/src/librmb/rados-cluster.h b/src/librmb/rados-cluster.h
@@ -60,6 +60,12 @@ class RadosCluster {
    * @return linux error code or 0 if successful
    *
    */
+
+  /**
+   * creates or returns the recovery index io ctx
+  */
+  virtual int recovery_index_io_ctx(const std::string &pool,librados::IoCtx *io_ctx) = 0;
+
   virtual int get_config_option(const char *option, std::string *value) = 0;
 
   /*!

diff --git a/src/librmb/rados-dovecot-ceph-cfg-impl.h b/src/librmb/rados-dovecot-ceph-cfg-impl.h
@@ -40,6 +40,7 @@ class RadosDovecotCephCfgImpl : public RadosDovecotCephCfg {
 
   int get_chunk_size() override { return std::stoi(dovecot_cfg.get_chunk_size());}
   std::string &get_pool_name() override { return dovecot_cfg.get_pool_name(); }
+  std::string &get_index_pool_name() override { return dovecot_cfg.get_index_pool_name(); };
 
   std::string &get_key_prefix_keywords() override { return dovecot_cfg.get_key_prefix_keywords(); }
   void update_metadata(const std::string &key, const char *value_) override {

diff --git a/src/librmb/rados-dovecot-ceph-cfg.h b/src/librmb/rados-dovecot-ceph-cfg.h
@@ -58,6 +58,8 @@ class RadosDovecotCephCfg {
   virtual std::map<std::string, std::string> *get_config() = 0;
 
   virtual std::string &get_pool_name() = 0;
+  virtual std::string &get_index_pool_name() = 0;
+
   virtual bool is_update_attributes() = 0;
 
   virtual void set_rbox_cfg_object_name(const std::string &value) = 0;

diff --git a/src/librmb/rados-dovecot-config.cpp b/src/librmb/rados-dovecot-config.cpp
@@ -19,6 +19,7 @@ namespace librmb {
 
 RadosConfig::RadosConfig()
     : pool_name("rbox_pool_name"),
+      index_pool_name("rbox_index_pool_name"),
       rbox_cfg_object_name("rbox_cfg_object_name"),
       rbox_cluster_name("rbox_cluster_name"),
       rados_username("rados_user_name"),
@@ -34,7 +35,7 @@ RadosConfig::RadosConfig()
       rbox_object_search_threads("rbox_object_search_threads") {
 
   config[pool_name] = "mail_storage";
-
+  config[index_pool_name] = "object_recovery";
   config[rbox_cfg_object_name] = "rbox_cfg";
   config[rbox_cluster_name] = "ceph";
   config[rados_username] = "client.admin";
@@ -75,6 +76,8 @@ void RadosConfig::update_metadata(const std::string &key, const char *value_) {
 std::string RadosConfig::to_string() {
   std::stringstream ss;
   ss << "Dovecot configuration: (90-plugin.conf)" << std::endl;
+  ss << "  " << pool_name << "=" << config[pool_name] << std::endl;
+  ss << "  " << index_pool_name << "=" << config[index_pool_name] << std::endl;
   ss << "  " << rbox_cfg_object_name << "=" << config[rbox_cfg_object_name] << std::endl;
   ss << "  " << rbox_cluster_name << "=" << config[rbox_cluster_name] << std::endl;
   ss << "  " << rados_username << "=" << config[rados_username] << std::endl;

diff --git a/src/librmb/rados-dovecot-config.h b/src/librmb/rados-dovecot-config.h
@@ -36,6 +36,8 @@ class RadosConfig {
   std::map<std::string, std::string> *get_config() { return &config; }
 
   std::string &get_pool_name() { return config[pool_name]; }
+  std::string &get_index_pool_name() { return config[index_pool_name]; };
+
   const std::string &get_rados_save_log_file() { return config[save_log]; }
   bool is_config_valid() { return is_valid; }
   void set_config_valid(bool is_valid_) { this->is_valid = is_valid_; }
@@ -79,7 +81,7 @@ class RadosConfig {
  private:
   std::map<std::string, std::string> config;
   std::string pool_name;
-
+  std::string index_pool_name;
   std::string rbox_cfg_object_name;
   std::string rbox_cluster_name;
   std::string rados_username;

diff --git a/src/librmb/rados-storage-impl.cpp b/src/librmb/rados-storage-impl.cpp
@@ -221,7 +221,7 @@ std::set<std::string> RadosStorageImpl::find_mails_async(const RadosMetadata *at
           uint32_t pseed;
           int r = sscanf(pg.c_str(), "%llu.%x", (long long unsigned *)&ppool, &pseed);
 
-          librados::NObjectIterator iter= io_ctx->nobjects_begin(pseed, filter);
+          librados::NObjectIterator iter= io_ctx->nobjects_begin(pseed);
           int count = 0;
           while (iter != librados::NObjectIterator::__EndObjectIterator) {
             std::string oid = iter->get_oid();
@@ -277,8 +277,10 @@ std::set<std::string> RadosStorageImpl::find_mails_async(const RadosMetadata *at
     return oid_list;
 }
 librados::IoCtx &RadosStorageImpl::get_io_ctx() { return io_ctx; }
+librados::IoCtx &RadosStorageImpl::get_recovery_io_ctx() { return recovery_io_ctx; }
 
-int RadosStorageImpl::open_connection(const std::string &poolname, const std::string &clustername,
+int RadosStorageImpl::open_connection(const std::string &poolname, const std::string &index_pool,
+                                      const std::string &clustername,
                                       const std::string &rados_username) {
   if (cluster->is_connected() && io_ctx_created) {
     // cluster is already connected!
@@ -288,22 +290,46 @@ int RadosStorageImpl::open_connection(const std::string &poolname, const std::st
   if (cluster->init(clustername, rados_username) < 0) {
     return -1;
   }
-  return create_connection(poolname);
+  return create_connection(poolname, index_pool);
+}
+int RadosStorageImpl::open_connection(const std::string &poolname,
+                                      const std::string &clustername,
+                                      const std::string &rados_username) {
+  if (cluster->is_connected() && io_ctx_created) {
+    // cluster is already connected!
+    return 1;
+  }
+
+  if (cluster->init(clustername, rados_username) < 0) {
+    return -1;
+  }
+  return create_connection(poolname, poolname);
+}
+int RadosStorageImpl::open_connection(const string &poolname, const string &index_pool) {
+  if (cluster->init() < 0) {
+    return -1;
+  }
+  return create_connection(poolname, index_pool);
 }
 
 int RadosStorageImpl::open_connection(const string &poolname) {
   if (cluster->init() < 0) {
     return -1;
   }
-  return create_connection(poolname);
+  return create_connection(poolname, poolname);
 }
 
-int RadosStorageImpl::create_connection(const std::string &poolname) {
+int RadosStorageImpl::create_connection(const std::string &poolname, const std::string &index_pool){
   // pool exists? else create
   int err = cluster->io_ctx_create(poolname, &io_ctx);
   if (err < 0) {
     return err;
   }
+
+  err = cluster->recovery_index_io_ctx(index_pool, &recovery_io_ctx);
+  if (err < 0) {
+    return err;
+  }
   string max_write_size_str;
   err = cluster->get_config_option(RadosStorageImpl::CFG_OSD_MAX_WRITE_SIZE, &max_write_size_str);
   if (err < 0) {
@@ -540,10 +566,53 @@ bool RadosStorageImpl::save_mail(RadosMail *mail) {
   }
   return save_mail(&write_op_xattr, mail);
 }
+
 librmb::RadosMail *RadosStorageImpl::alloc_rados_mail() { return new librmb::RadosMail(); }
+
 void RadosStorageImpl::free_rados_mail(librmb::RadosMail *mail) {
   if (mail != nullptr) {
     delete mail;
     mail = nullptr;
   }
 }
+
+int RadosStorageImpl::ceph_index_append(const std::string &oid) {  
+  librados::bufferlist bl;
+  bl.append(RadosUtils::convert_to_ceph_index(oid));
+  return get_recovery_io_ctx().append( get_namespace(),bl, bl.length());
+}
+
+int RadosStorageImpl::ceph_index_append(const std::set<std::string> &oids) {
+  librados::bufferlist bl;
+  bl.append(RadosUtils::convert_to_ceph_index(oids));
+  return get_recovery_io_ctx().append( get_namespace(),bl, bl.length());
+}
+int RadosStorageImpl::ceph_index_overwrite(const std::set<std::string> &oids) {
+  librados::bufferlist bl;
+  bl.append(RadosUtils::convert_to_ceph_index(oids));
+  return get_recovery_io_ctx().write_full( get_namespace(),bl);
+}
+std::set<std::string> RadosStorageImpl::ceph_index_read() {
+  std::set<std::string> index;
+  librados::bufferlist bl;
+  size_t max = INT_MAX;
+  int64_t psize;
+  time_t pmtime;
+  get_recovery_io_ctx().stat(get_namespace(), &psize, &pmtime);
+  if(psize <=0){
+    return index;
+  }
+  //std::cout << " NAMESPACE: " << get_namespace() << " exist? " << exist << " size : " << psize << std::endl;
+  int ret = get_recovery_io_ctx().read(get_namespace(),bl, max,0);
+
+
+  if(ret < 0){
+    return index;
+  }
+  index = RadosUtils::ceph_index_to_set(bl.c_str());
+  return index;
+}
+int RadosStorageImpl::ceph_index_delete() {
+  return get_recovery_io_ctx().remove(get_namespace());
+}
+
diff --git a/src/librmb/rados-storage-impl.h b/src/librmb/rados-storage-impl.h
@@ -33,6 +33,8 @@ class RadosStorageImpl : public RadosStorage {
   virtual ~RadosStorageImpl();
 
   librados::IoCtx &get_io_ctx() override;
+  librados::IoCtx &get_recovery_io_ctx() override;
+
   int stat_mail(const std::string &oid, uint64_t *psize, time_t *pmtime) override;
   void set_namespace(const std::string &_nspace) override;
   std::string get_namespace() override { return nspace; }
@@ -56,6 +58,12 @@ class RadosStorageImpl : public RadosStorage {
   std::set<std::string> find_mails_async(const RadosMetadata *attr, std::string &pool_name, int num_threads, void (*ptr)(std::string&)) override;
 
   int open_connection(const std::string &poolname) override;
+  int open_connection(const std::string &poolname, const std::string &index_pool) override;
+
+
+  int open_connection(const std::string &poolname, const std::string &index_pool,
+                      const std::string &clustername,
+                      const std::string &rados_username) override;
   int open_connection(const std::string &poolname, const std::string &clustername,
                       const std::string &rados_username) override;
   void close_connection() override;
@@ -77,16 +85,23 @@ class RadosStorageImpl : public RadosStorage {
 
   void free_rados_mail(librmb::RadosMail *mail) override;
 
-
+  int ceph_index_append(const std::string &oid)  override;
+  int ceph_index_append(const std::set<std::string> &oids)  override;
+  int ceph_index_overwrite(const std::set<std::string> &oids)  override;
+  std::set<std::string> ceph_index_read() override;
+  int ceph_index_delete() override;
+
  private:
-  int create_connection(const std::string &poolname);
+  int create_connection(const std::string &poolname,const std::string &index_pool);
 
  private:
   RadosCluster *cluster;
   int max_write_size;
   int max_object_size;
   std::string nspace;
   librados::IoCtx io_ctx;
+  librados::IoCtx recovery_io_ctx;
+
   bool io_ctx_created;
   std::string pool_name;
   enum rbox_ceph_aio_wait_method wait_method;