✨ feat: 支持用户自行定义 embedding model

[cookieY]

💻 变更类型 | Change Type

• ✨ feat
• 🐛 fix
• ♻️ refactor
• 💄 style
• 👷 build
• ⚡️ perf
• 📝 docs
• 🔨 chore

🔀 变更说明 | Description of Change

本次变更主要实现了embedding 模型可配置性

通过新增环境变量DEFAULT_EMBEDDING_MODEL实现可自主配置embedding 模型, provider基于现有模型供应商列表

示例:

以 / 为分隔符 openai 为模型提供商, text-embedding-3-small为 embedding 模型

DEFAULT_EMBEDDING_MODEL=openai/text-embedding-3-small   

📝 补充信息 | Additional Information

目前已支持 openai / bedrock / ollama / zhipu模型提供商 的 embedding 模型

可在agent-runtime下对各模型提供商实现 embeddings 方法从而增加 embedding 模型支持

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The index object attribute is stored in the data table

Index and object should not be stored in the data table, right? Is there an implementation problem?

From what I have seen so far, it seems that these two values ​​​​do not have any special logical processing. If there is no special logical processing for these two attributes, I think it is best to extract them. At present, ollama bedrock does not have these two attributes. However, in order to be compatible with existing methods, the packaging needs to be traversed again, which is a waste of performance.

Then I suggest that it is better to directly return the array of embedding, without the need for the two fields object and index. It is recommended that you first make a refactor PR to transform the existing implementation into a version without object and index.

[cookieY]

@arvinxx 目前已搞定
DEFAULT_FILES_CONFIG="embedding_model=openai/embedding-text-3-large,reranker_model=cohere/rerank-english-v3.0,query_mode=full_text" 格式的需求。当用户不填写时默认使用 openai/text-embedding-3-small

目前代码个人觉得有些地方想看下有没有更好的优化方案，目前是通过在embeddingChunks，runRecordEvaluation，semanticSearch，semanticSearchForChat方法中添加以下方法获得 embedding model 信息，是否有更好的优化方案

const model =getServerGlobalConfig().defaultEmbed?.embedding_model?.model DEFAULT_EMBEDDING_MODEL.model;
const provider =getServerGlobalConfig().defaultEmbed?.embedding_model?.provider ??DEFAULT_EMBEDDING_MODEL.provider;

2. 输出格式实现逻辑中有一段

 const items: NewEmbeddingsItem[] =
                  embeddings?.map((e) => ({
                    chunkId: chunks[e.index].id,
                    embeddings: e.embedding,
                    fileId: input.fileId,
                    model: model,
                  })) || [];

目前是通过embedding 返回数据中的 index 作为索引查找，当前情况下 不管是更改格式还是不更改，都会出现一部分提供商返回的数据需要进行二次包装。(不改，bedrock/ollama需要包装。改，zhipu/openai 需要包装)，我的建议是暂时先不动了

[arvinxx]

@cookieY rebase下重新提交吧

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@cookieY Rebase and resubmit

[cookieY]

@arvinxx 等等我 reset 后重新提交一个新的分支吧

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@arvinxx Wait for me to reset and resubmit a new branch.

[cookieY]

@arvinxx #4370

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@arvinxx #4370