-
Notifications
You must be signed in to change notification settings - Fork 418
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
…3416) * Initial work for Qdrant resource and component (#3131) This PR adds a Resource and initial Component (plus playground and tests) Resource exposes two API access endpoints (REST + gRPC). The QdrantClient for .NET uses gRPC by default but Semantic Kernel does not use that client library so the other exposed endpoint is helpful. REST endpoint by default exposes a Web UI dashboard -- this is excluded in Publish (per documentation in source, confirmed with Qdrant contributors). Endpoints are secured non-optionally with an API Key (using ParameterResource -- will generate random if not provided). Component used QdrantClient .NET version (recommended from Qdrant). This component uses gRPC endpoint by default. Settings will read from standard env vars for ConnectionString + API key or from config for the component itself if provided. * Initial draft of QdrantServerResource - exposed gRPC endpoint as primary (c# client uses that) - exposes HTTP endpoint optionally (as dashboard is there) - defaults to secure the access via API key (mainly due to dashboard) * Fix-up after main rebase on removing InputAnnotation * Remove dashboard in Publish mode - change how dashboard is removed in Publish (per docs) - add 'qdrant' to spelling.dic - fix tests * Fixup code style violations * Change ApiKey reference - Add WithReference overload (setting cn string + key) - Added tests for named parameter on manifest - Changed playground app * Initial Qdrant component work - adds component (using Qdrant.QdrantClient) - Change playground to use DI component * Changed rest endpoint name - changed to 'rest' as named endpoint - added to reference as a ConnectionString_{name}_rest (for support for SK) - fixed tests * Addressing PR feedback - Changed parameters to component config - change playground app to keyed services - changed playground to use shared servicedefaults * Fixing volume mounts to correct target location * Add missing README to component * Adding logo usage comment to readme after permission from Andrey V from Qdrant * Changed playground sample - matching sample Qdrant/.NET workbook sample * PR Feedback: Name maps to client name used * PR feedback on connection sring - Moved to Endpoint/Key connection string - Renamed to Key - Modified schema to match/updated tests - Changed primary endpoint to use scheme instead of hardcode - removed env var for API key only - fixed renamed component proj location in sln * Updating readme/comments to match config * Endpoint property typo fix * PR Feedback - No need for endpoint null check - Fixed tests * Cleanup of ServiceDefaults Change to ReferenceExpression.Create * More PR Feedback - Rename component correctly - Add component client tests - Add component logging (and document) - Add property for 2nd endpoint on resource directly * PR feedback * Fix up the playground to run with latest changes. * Rename QdrantSettings to QdrantClientSettings to match the pattern in other components. Add class level summary docs. --------- Co-authored-by: Eric Erhardt <[email protected]> * Addressing port/endpoint issues on Qdrant (#3422) * Addressing port/endpoint issues on Qdrant * PR feedback (param name) * Fix up tests * Respond to PR feedback. Rename ports and endpoint names to be consistent. * PR feedback - moved to using grpc/http endpoint name consistent with config and prior art - fixed wrong link in readme - fixed/validated resource hosting test --------- Co-authored-by: Eric Erhardt <[email protected]> * Fix WithReference extension return type (#3449) * Fix Qdrant WithReference to allow any IResourceWithEnvironment. Also fix the same pattern in AWS. --------- Co-authored-by: Tim Heuer <[email protected]>
- Loading branch information
Showing
37 changed files
with
1,297 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| using Qdrant.Client; | ||
| using Qdrant.Client.Grpc; | ||
|
|
||
| var builder = WebApplication.CreateBuilder(args); | ||
|
|
||
| // Add service defaults & Aspire components. | ||
| builder.AddServiceDefaults(); | ||
|
|
||
| // Add services to the container. | ||
| builder.Services.AddProblemDetails(); | ||
|
|
||
| builder.AddQdrantClient("qdrant"); | ||
|
|
||
| var app = builder.Build(); | ||
|
|
||
| // Configure the HTTP request pipeline. | ||
| app.UseExceptionHandler(); | ||
|
|
||
| app.MapGet("/create", async (QdrantClient client, ILogger<Program> logger) => | ||
| { | ||
| var collections = await client.ListCollectionsAsync(); | ||
| if (collections.Any(x => x.Contains("movie_collection"))) | ||
| { | ||
| await client.DeleteCollectionAsync("movie_collection"); | ||
| } | ||
| await client.CreateCollectionAsync("movie_collection", new VectorParams { Size = 2, Distance = Distance.Cosine }); | ||
| var collectionInfo = await client.GetCollectionInfoAsync("movie_collection"); | ||
| logger.LogInformation(collectionInfo.ToString()); | ||
| // generate some vectors | ||
| var data = new[] | ||
| { | ||
| new PointStruct | ||
| { | ||
| Id = 1, | ||
| Vectors = new [] {0.10022575f, -0.23998135f}, | ||
| Payload = | ||
| { | ||
| ["title"] = "The Lion King" | ||
| } | ||
| }, | ||
| new PointStruct | ||
| { | ||
| Id = 2, | ||
| Vectors = new [] {0.10327095f, 0.2563685f}, | ||
| Payload = | ||
| { | ||
| ["title"] = "Inception" | ||
| } | ||
| }, | ||
| new PointStruct | ||
| { | ||
| Id = 3, | ||
| Vectors = new [] {0.095857024f, -0.201278f}, | ||
| Payload = | ||
| { | ||
| ["title"] = "Toy Story" | ||
| } | ||
| }, | ||
| new PointStruct | ||
| { | ||
| Id = 4, | ||
| Vectors = new [] {0.106827796f, 0.21676421f}, | ||
| Payload = | ||
| { | ||
| ["title"] = "Pulp Function" | ||
| } | ||
| }, | ||
| new PointStruct | ||
| { | ||
| Id = 5, | ||
| Vectors = new [] {0.09568083f, -0.21177962f}, | ||
| Payload = | ||
| { | ||
| ["title"] = "Shrek" | ||
| } | ||
| }, | ||
| }; | ||
| var updateResult = await client.UpsertAsync("movie_collection", data); | ||
| return updateResult.Status; | ||
| }); | ||
|
|
||
| app.MapGet("/search", async (QdrantClient client) => | ||
| { | ||
| var results = await client.SearchAsync("movie_collection", new[] { 0.12217915f, -0.034832448f }, limit: 3); | ||
| return results.Select(titles => titles.Payload["title"].StringValue); | ||
| }); | ||
|
|
||
| app.MapDefaultEndpoints(); | ||
|
|
||
| app.Run(); |
25 changes: 25 additions & 0 deletions
25
playground/Qdrant/Qdrant.ApiService/Properties/launchSettings.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| { | ||
| "$schema": "https://json.schemastore.org/launchsettings.json", | ||
| "profiles": { | ||
| "https": { | ||
| "commandName": "Project", | ||
| "dotnetRunMessages": true, | ||
| "launchBrowser": true, | ||
| "launchUrl": "", | ||
| "applicationUrl": "https://localhost:5450;http://localhost:5451", | ||
| "environmentVariables": { | ||
| "ASPNETCORE_ENVIRONMENT": "Development" | ||
| } | ||
| }, | ||
| "http": { | ||
| "commandName": "Project", | ||
| "dotnetRunMessages": true, | ||
| "launchBrowser": true, | ||
| "launchUrl": "", | ||
| "applicationUrl": "http://localhost:5451", | ||
| "environmentVariables": { | ||
| "ASPNETCORE_ENVIRONMENT": "Development" | ||
| } | ||
| } | ||
| } | ||
| } |
15 changes: 15 additions & 0 deletions
15
playground/Qdrant/Qdrant.ApiService/Qdrant.ApiService.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk.Web"> | ||
|
|
||
| <PropertyGroup> | ||
| <TargetFramework>net8.0</TargetFramework> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <UserSecretsId>1e49caab-af46-4c24-8011-953ec12b4069</UserSecretsId> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\..\src\Components\Aspire.Qdrant.Client\Aspire.Qdrant.Client.csproj" /> | ||
| <ProjectReference Include="..\..\Playground.ServiceDefaults\Playground.ServiceDefaults.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
8 changes: 8 additions & 0 deletions
8
playground/Qdrant/Qdrant.ApiService/appsettings.Development.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| { | ||
| "Logging": { | ||
| "LogLevel": { | ||
| "Default": "Information", | ||
| "Microsoft.AspNetCore": "Warning" | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| { | ||
| "Logging": { | ||
| "LogLevel": { | ||
| "Default": "Information", | ||
| "Microsoft.AspNetCore": "Warning", | ||
| "Qdrant.Client": "Debug" | ||
| } | ||
| }, | ||
| "AllowedHosts": "*" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| <Project> | ||
|
|
||
| <Import Project="$([MSBuild]::GetPathOfFileAbove('Directory.Build.props', '$(MSBuildThisFileDirectory)../'))" /> | ||
|
|
||
| <!-- NOTE: This line is only required because we are using P2P references, not NuGet. It will not exist in real apps. --> | ||
| <Import Project="../../../src/Aspire.Hosting.AppHost/build/Aspire.Hosting.AppHost.props" /> | ||
|
|
||
| </Project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| <Project> | ||
|
|
||
| <Import Project="$([MSBuild]::GetPathOfFileAbove('Directory.Build.targets', '$(MSBuildThisFileDirectory)../'))" /> | ||
|
|
||
| <!-- NOTE: These lines are only required because we are using P2P references, not NuGet. They will not exist in real apps. --> | ||
| <Import Project="..\..\..\src\Aspire.Hosting.AppHost\build\Aspire.Hosting.AppHost.targets" /> | ||
| <Import Project="..\..\..\src\Aspire.Hosting.Sdk\SDK\Sdk.targets" /> | ||
|
|
||
| </Project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
|
|
||
| var builder = DistributedApplication.CreateBuilder(args); | ||
|
|
||
| var qdrant = builder.AddQdrant("qdrant") | ||
| .WithDataVolume("qdrant_data"); | ||
|
|
||
| builder.AddProject<Projects.Qdrant_ApiService>("apiservice") | ||
| .WithReference(qdrant); | ||
|
|
||
| builder.Build().Run(); |
32 changes: 32 additions & 0 deletions
32
playground/Qdrant/Qdrant.AppHost/Properties/launchSettings.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| { | ||
| "$schema": "https://json.schemastore.org/launchsettings.json", | ||
| "profiles": { | ||
| "https": { | ||
| "commandName": "Project", | ||
| "dotnetRunMessages": true, | ||
| "launchBrowser": true, | ||
| "applicationUrl": "https://localhost:15206;http://localhost:15207", | ||
| "environmentVariables": { | ||
| "ASPNETCORE_ENVIRONMENT": "Development", | ||
| "DOTNET_ENVIRONMENT": "Development", | ||
| "DOTNET_DASHBOARD_OTLP_ENDPOINT_URL": "https://localhost:16022", | ||
| "DOTNET_RESOURCE_SERVICE_ENDPOINT_URL": "https://localhost:17038", | ||
| "DOTNET_ASPIRE_SHOW_DASHBOARD_RESOURCES": "true" | ||
| } | ||
| }, | ||
| "http": { | ||
| "commandName": "Project", | ||
| "dotnetRunMessages": true, | ||
| "launchBrowser": true, | ||
| "applicationUrl": "http://localhost:15207", | ||
| "environmentVariables": { | ||
| "ASPNETCORE_ENVIRONMENT": "Development", | ||
| "DOTNET_ENVIRONMENT": "Development", | ||
| "DOTNET_DASHBOARD_OTLP_ENDPOINT_URL": "http://localhost:16022", | ||
| "DOTNET_RESOURCE_SERVICE_ENDPOINT_URL": "http://localhost:17039", | ||
| "DOTNET_ASPIRE_SHOW_DASHBOARD_RESOURCES": "true", | ||
| "ASPIRE_ALLOW_UNSECURED_TRANSPORT": "true" | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <OutputType>Exe</OutputType> | ||
| <TargetFramework>net8.0</TargetFramework> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <IsAspireHost>true</IsAspireHost> | ||
| <UserSecretsId>10c36641-05e0-4bfb-ad9d-a588431430f0</UserSecretsId> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <Compile Include="$(SharedDir)KnownResourceNames.cs" Link="KnownResourceNames.cs" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\..\src\Aspire.Dashboard\Aspire.Dashboard.csproj" /> | ||
| <ProjectReference Include="..\..\..\src\Aspire.Hosting.AppHost\Aspire.Hosting.AppHost.csproj" IsAspireProjectResource="false" /> | ||
| <ProjectReference Include="..\..\..\src\Aspire.Hosting.Qdrant\Aspire.Hosting.Qdrant.csproj" IsAspireProjectResource="false" /> | ||
| <ProjectReference Include="..\Qdrant.ApiService\Qdrant.ApiService.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
8 changes: 8 additions & 0 deletions
8
playground/Qdrant/Qdrant.AppHost/appsettings.Development.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| { | ||
| "Logging": { | ||
| "LogLevel": { | ||
| "Default": "Information", | ||
| "Microsoft.AspNetCore": "Warning" | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "Logging": { | ||
| "LogLevel": { | ||
| "Default": "Information", | ||
| "Microsoft.AspNetCore": "Warning", | ||
| "Aspire.Hosting.Dcp": "Warning" | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -45,6 +45,7 @@ pgadmin | |
| postgre | ||
| postgres | ||
| protoc | ||
| qdrant | ||
| rabbitmq | ||
| redis | ||
| rediscommander | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.