Implement embeddings for use with LLM agents

[ahuang11]

Uses duckdb instead of chromadb. Currently subsets embeddings by table name. Not sure how to handle ephemeral tables yet.

import io
import asyncio

import lumen.ai as lmai
import panel as pn
import pandas as pd

from lumen.sources.duckdb import DuckDBSource
from lumen.ai.analysis import Join

pn.extension(
    "tabulator",
    "codeeditor",
    "vega",
    inline=False,
    template="fast",
    notifications=True,
)

llm = lmai.llm.OpenAI(
    # interceptor_path="interceptor.db"
)

embeddings = lmai.embeddings.OpenAIEmbeddings.from_dict(
    {
        "windturbines.parquet": [
            "The creator of this dataset is named Andrew HH",
            "To run the imaginary, test analysis, simply divide turbine count by 3",
        ]
    }
)

cs = lmai.memory["current_source"] = DuckDBSource(
    tables=["windturbines.parquet"],
    uri=":memory:",
    initializers=["INSTALL httpfs;", "LOAD httpfs;"],
)

custom_agent = lmai.agents.AnalysisAgent(analyses=[Join])

assistant = lmai.Assistant(
    llm=llm,
    agents=[
        lmai.agents.SourceAgent,
        lmai.agents.TableAgent,
        lmai.agents.TableListAgent,
        lmai.agents.SQLAgent,
        lmai.agents.PipelineAgent,
        lmai.agents.hvPlotAgent,
        lmai.agents.ChatAgent,
        custom_agent,
    ],
    embeddings=embeddings,
    suggestions=["Who is the creator of this dataset?"],
)
assistant.servable("Lumen.ai")
assistant.controls().servable(area="sidebar")

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 69 lines in your changes missing coverage. Please review.

Project coverage is 56.40%. Comparing base (2622c0b) to head (58b32fd).
Report is 19 commits behind head on main.

Files	Patch %	Lines
lumen/ai/embeddings.py	0.00%	57 Missing ⚠️
lumen/ai/agents.py	0.00%	6 Missing ⚠️
lumen/ai/assistant.py	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #680      +/-   ##
==========================================
- Coverage   57.03%   56.40%   -0.64%     
==========================================
  Files          98       98              
  Lines       11598    11737     +139     
==========================================
+ Hits         6615     6620       +5     
- Misses       4983     5117     +134     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ahuang11]

Another TODO: handle ephemeral tables