SessionPool how to change enable shared between threads.

[stevezhou6]

SessionPool stores all thread's HTTPSessionBase objects. and ervery can get HTTPSessionBase frome SessionPool .

[afrind]

We should probably add this an example with some documentation. The basic idea is that a SessionPool object should only be thread local, and you use a ServerIdleSessionController to assist with transferring sessions between session pools.

Here I extracted our routine for getting a session (slightly modified), including transferring idle sessions between threads. You will need some other thread local structures to make attachThreadLocals work properly. It's a little clunky and this should be better encapsulated, but it works for us.

class Server {
  public:
    folly::ThreadLocal<SessionPool> sessionPool_;
    ServerIdleSessionController serverIdleSessionController_;
 
    //  Get a transaction on a pooled connection, or nullptr if none are available
   folly::Future<HTTPTransaction*> Server::getTransaction(
    HTTPTransaction::Handler* handler,
    folly::EventBase* eventBase,
    HTTPUpstreamSessionController* controller);
};

folly::Future<HTTPTransaction*> Server::getTransaction(
    HTTPTransaction::Handler* handler,
    folly::EventBase* eventBase,
    HTTPUpstreamSessionController* controller) {
  // Get thread-local session pool.
  HTTPTransaction* retTxn = nullptr;
  auto pool = sessionPool_.get();
  if (pool) {
    // Try to get a transaction from current thread.
    retTxn = pool->getTransaction(handler);
    // If nothing is available, try transferring from another thread.
    if (!retTxn) {
      auto future = serverIdleSessionController_.getIdleSession();
      auto func = [=](HTTPSessionBase* session) {
        HTTPTransaction* txn = nullptr;
        if (session) {
          auto evb = session->getEventBase();
          if (!evb || !evb->isInEventBaseThread()) {
            session->attachThreadLocals(
                eventBase,
                /*sslContext*/=nullptr,
                /*timeoutSet=*/,
                /*sessionStats=*/nullptr,
                filterFn,
                /*codecStats=*/nullptr,
                controller);
            /*
               auto cm = getThreadLocalUpstreamConnectionManager();
               if (cm) {
               cm->addConnection(session, true);
               }
            */
          }
          pool->putSession(session);
          txn = pool->getTransaction(handler);
        }
        return txn;
      };
      if (future.isReady()) {
        // Either getIdleSession() returned null or the other thread was faster.
        HTTPSessionBase* session = future.value();
        if (!session) {
          return folly::makeFuture<HTTPTransaction*>(nullptr);
        } else if (session->getEventBase()) {
          // It's impossible to get an idle session from current thread.
          // We already checked it at the beginning.
          CHECK(!session->getEventBase()->isInEventBaseThread());
        }
      }
      return std::move(future)
          .via(eventBase)
          .thenValue(func)
          // Erase the executor because executor from getIdleSession completes
          // inline and has a null executor
          .semi()
          .toUnsafeFuture();
    }
  }
  VLOG(4) << "Server " << *this << " got transaction=" << retTxn
          << " from pool " << pool;
  return folly::makeFuture<HTTPTransaction*>((HTTPTransaction*)retTxn);
}

[stevezhou6]

Thank you @afrind that helps a lot :)
i want to ask how to use CPUThreadPoolExecutor and folly::ThreadLocal sessionPool_

class Request {
 private:
  std::unique_ptr<proxygen::HTTPMessage> headers_;
  std::unique_ptr<folly::IOBuf> body_;
};

class Server {
public:
  folly::ThreadLocal<SessionPool> sessionPool_;
  ServerIdleSessionController serverIdleSessionController_;
  folly::Future<HTTPTransaction*> Server::getTransaction(
      HTTPTransaction::Handler* handler,
      folly::EventBase* eventBase,
      HTTPUpstreamSessionController* controller);
  void Process(Request* req);
};
void httpRequestTask(Request* req) {
	//how to get current Server folly::ThreadLocal<SessionPool>
}
void main() {
  // i want to use CPUThreadPoolExecutor process httprequest task

  //every Server get one thread
  //ThreadFactory  how to combin folly::ThreadLocal<SessionPool> to every thread
  // one thread one Server
  auto ThreadPool = std::make_shared<folly::CPUThreadPoolExecutor>(3);
  // CPUThreadPoolExecutor  create 3 Server
  Request req1;
  Request req2;
  ThreadPool->add(std::bind(&httpRequestTask, &req1));
  ThreadPool->add(std::bind(&httpRequestTask, &req2));
  // ...
  ThreadPool->join();
}

[afrind]

You can't really use HTTPUpstreamSessions in a CPUThreadPool. You need to use them in an IOThreadPool, or at least some thread that has an EventBase that loops.

If you have a mixture of CPU heavy work and IO work, then you app needs to coordinate between them.

Though for your specific question, see the documentation for folly::ThreadLocal - you can just call server.sessionPool_.get() to get or create the current thread's SessionPool I think.

[nidhidamodaran]

@afrind Is there a performance impact on using these changes compared to non shared sessionpool ?

[afrind]

Depending on the number of threads you have, and the number of concurrent requests bound for the same server, using a idle session controller can be a big memory win. eg if you have 1000 servers that allow 60s of idle time, and a client/proxy that has 50 threads, but a per-server request rate of 1/s, using the idle controller will reduce you from 50k connections to 50. You will also find that your connection reuse rate is better, since it's possible some threads have 0 requests fo a given server in 60s and the connection would idle out.

Further, each connection will get more use and spend more time with the optimal CWND. It's possible there could be a tiny amount of latency to transfer the connection from one thread to another, but it should be a matter of microseconds.