Feature: Token count for vectors

[ManishMadan2882]

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
  • Closes 🚀 Feature: Token count for added documents #947
  • Calculates token count for document and remote workers.
  • Appends token count in GET /api/combine as

[
    {
        "date": "24/05/2024 20:25:12",
        "description": "Demo",
        "docLink": "/home/manish/Documents/workplace/docsgpt/application/indexes/local/LLM",
        "fullName": "Demo,
        "language": "LLM",
        "location": "local",
        "model": "huggingface_sentence-transformers/all-mpnet-base-v2",
        "name": "Demo",
        "tokens": "0",
        "version": ""
    }, .... 
]

• Frontend : Shows token usage column for private vectors in Settings > Documents.
• Frontend: Updates Doc type in misc.ts : optional key tokens

• Why was this change needed? (You can also link to an open issue here)
  • More transparency in terms of tokens expenditure for each document or remote vector
• Other information: Settings > Docs
  

[vercel]

@ManishMadan2882 is attempting to deploy a commit to the Arc53 Team on Vercel.

A member of the Team first needs to authorize it.

[dartpain]

Perfect, thank you!