Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Search] Move GA-Related Changes from Feature Branch to Main #39813

Merged
merged 2 commits into from
Nov 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion sdk/search/Azure.Search.Documents/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Release History

## 11.5.0-beta.6 (Unreleased)
## 11.5.0 (Unreleased)

### Features Added

Expand Down
4 changes: 3 additions & 1 deletion sdk/search/Azure.Search.Documents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,9 @@ Semantic search enhances the quality of search results for text-based queries. B
- It applies secondary ranking to the initial result set, promoting the most semantically relevant results to the top.
- It extracts and returns captions and answers in the response, which can be displayed on a search page to enhance the user's search experience.

To learn more about Semantic Search, you can refer to the [documentation](https://learn.microsoft.com/azure/search/vector-search-overview).
To learn more about Semantic Search, you can refer to the [sample](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample08_SemanticSearch.md).

Additionally, for more comprehensive information about Semantic Search, including its concepts and usage, you can refer to the [documentation](https://learn.microsoft.com/azure/search/semantic-search-overview). The documentation provides in-depth explanations and guidance on leveraging the power of Semantic Search in Azure Cognitive Search.

#### Vector Search

Expand Down

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion sdk/search/Azure.Search.Documents/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "net",
"TagPrefix": "net/search/Azure.Search.Documents",
"Tag": "net/search/Azure.Search.Documents_547641209a"
"Tag": "net/search/Azure.Search.Documents_c3ad072221"
}
1 change: 1 addition & 0 deletions sdk/search/Azure.Search.Documents/samples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,4 @@ description: Samples for the Azure.Search.Documents client library
- Learn about different ways to [index documents](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample05_IndexingDocuments.md).
- Use your own Key Vault key to [encrypt your data source connections, skillsets, indexers, and index](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample06_EncryptedIndex.md).
- Learn how to [index vector fields and perform vector search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch.md).
- Learn how to [perform semantic search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample08_SemanticSearch.md).
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,6 @@ SearchIndex index = new SearchIndex("movies")
SearchableField genreField = new SearchableField("genre")
{
AnalyzerName = LexicalAnalyzerName.Values.EnLucene,
NormalizerName = LexicalNormalizerName.Lowercase,
IsFacetable = true,
IsFilterable = true
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,11 @@ Please refer the [documentation](https://learn.microsoft.com/azure/search/vector

Here's the list of samples that will show you how to index the vector fields and perform vector search using .NET SDK.

* [Vector Search Using RAW Vector Query](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingRawVectorQuery.md#vector-search-using-raw-vector-query)
* [Single Vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingRawVectorQuery.md#single-vector-search)
* [Single Vector Search With Filter](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingRawVectorQuery.md#single-vector-search-with-filter)
* [Hybrid Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingRawVectorQuery.md#hybrid-search)
* [Multi-vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingRawVectorQuery.md#multi-vector-search)
* [Multi-field Vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingRawVectorQuery.md#multi-field-vector-search)
* [Vector Search Using Vectorizable Text Query](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizableTextQuery.md#vector-search-using-vectorizable-text-query)
* [Single Vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizableTextQuery.md#single-vector-search)
* [Single Vector Search With Filter](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizableTextQuery.md#single-vector-search-with-filter)
* [Hybrid Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizableTextQuery.md#hybrid-search)
* [Multi-vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizableTextQuery.md#multi-vector-search)
* [Multi-field Vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizableTextQuery.md#multi-field-vector-search)
* [Vector Search Using Vectorized Query](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizedQuery.md#vector-search-using-vector-query)
* [Single Vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizedQuery.md#single-vector-search)
* [Single Vector Search With Filter](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizedQuery.md#single-vector-search-with-filter)
* [Hybrid Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizedQuery.md#hybrid-search)
* [Multi-Vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizedQuery.md#multi-vector-search)
* [Multi-Field Vector Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizedQuery.md#multi-field-vector-search)
* [Vector Semantic Hybrid Search](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingSemanticHybridQuery.md)
* [Vector Search Using Field Builder](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingFieldBuilder.md)
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The `FieldBuilder` class allows you to define a Search index from a model type.

## Model Creation

Consider the following model, which includes a property named `DescriptionVector` that represents a vector field. To configure a vector field, you must provide the model dimensions, indicating the size of the embeddings generated for this field, as well as the name of the vector search profile that specifies the algorithm configuration and any optional parameters for searching the vector field.
Consider the following model, which includes a property named `DescriptionVector` that represents a vector field. To configure a vector field, you must provide the model dimensions, indicating the size of the embeddings generated for this field, as well as the name of the vector search profile that specifies the algorithm configuration in the `VectorSearchField` attribute.

```C# Snippet:Azure_Search_Documents_Tests_Samples_Sample07_Vector_Search_FieldBuilder_Model
public class MyDocument
Expand All @@ -18,7 +18,7 @@ public class MyDocument
[SearchableField(AnalyzerName = "en.microsoft")]
public string Description { get; set; }

[SearchableField(VectorSearchDimensions = "1536", VectorSearchProfile = "my-vector-profile")]
[VectorSearchField(VectorSearchDimensions = 1536, VectorSearchProfileName = "my-vector-profile")]
public IReadOnlyList<float> DescriptionVector { get; set; }
}
```
Expand All @@ -28,7 +28,7 @@ public class MyDocument
We will create an instace of `SearchIndex` and use `FieldBuilder` to define fields based on the `MyDocument` model class.

```C# Snippet:Azure_Search_Documents_Tests_Samples_Sample07_Vector_Search_Index_UsingFieldBuilder
string vectorSearchProfile = "my-vector-profile";
string vectorSearchProfileName = "my-vector-profile";
string vectorSearchHnswConfig = "my-hsnw-vector-config";

string indexName = "MyDocument";
Expand All @@ -40,11 +40,11 @@ SearchIndex searchIndex = new SearchIndex(indexName)
{
Profiles =
{
new VectorSearchProfile(vectorSearchProfile, vectorSearchHnswConfig)
new VectorSearchProfile(vectorSearchProfileName, vectorSearchHnswConfig)
},
Algorithms =
{
new HnswVectorSearchAlgorithmConfiguration(vectorSearchHnswConfig)
new HnswAlgorithmConfiguration(vectorSearchHnswConfig)
}
},
};
Expand All @@ -61,4 +61,4 @@ SearchIndexClient indexClient = new(endpoint, credential);
await indexClient.CreateIndexAsync(searchIndex);
```

To perform vector search please refer to the [Vector Search Using RAW Vector Query](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingRawVectorQuery.md) or [Vector Search Using Vectorizable Text Query](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizableTextQuery.md) samples.
To perform vector search please refer to the [Vector Search Using Vector Query](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/search/Azure.Search.Documents/samples/Sample07_VectorSearch_UsingVectorizedQuery.md) sample.
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Let's consider the example of a `Hotel`. First, we need to create an index for s
We will create an instace of `SearchIndex` and define `Hotel` fields.

```C# Snippet:Azure_Search_Documents_Tests_Samples_Sample07_Vector_Semantic_Hybrid_Search_Index
string vectorSearchProfile = "my-vector-profile";
string vectorSearchProfileName = "my-vector-profile";
string vectorSearchHnswConfig = "my-hsnw-vector-config";
int modelDimensions = 1536;

Expand All @@ -21,47 +21,37 @@ SearchIndex searchIndex = new(indexName)
new SimpleField("HotelId", SearchFieldDataType.String) { IsKey = true, IsFilterable = true, IsSortable = true, IsFacetable = true },
new SearchableField("HotelName") { IsFilterable = true, IsSortable = true },
new SearchableField("Description") { IsFilterable = true },
new SearchField("DescriptionVector", SearchFieldDataType.Collection(SearchFieldDataType.Single))
{
IsSearchable = true,
VectorSearchDimensions = modelDimensions,
VectorSearchProfile = vectorSearchProfile
},
new VectorSearchField("DescriptionVector", modelDimensions, vectorSearchProfileName),
new SearchableField("Category") { IsFilterable = true, IsSortable = true, IsFacetable = true },
new SearchField("CategoryVector", SearchFieldDataType.Collection(SearchFieldDataType.Single))
{
IsSearchable = true,
VectorSearchDimensions = modelDimensions,
VectorSearchProfile = vectorSearchProfile
},
new VectorSearchField("CategoryVector", modelDimensions, vectorSearchProfileName),
},
VectorSearch = new()
{
Profiles =
{
new VectorSearchProfile(vectorSearchProfile, vectorSearchHnswConfig)
new VectorSearchProfile(vectorSearchProfileName, vectorSearchHnswConfig)
},
Algorithms =
{
new HnswVectorSearchAlgorithmConfiguration(vectorSearchHnswConfig)
new HnswAlgorithmConfiguration(vectorSearchHnswConfig)
}
},
SemanticSettings = new()
SemanticSearch = new()
{
Configurations =
{
new SemanticConfiguration("my-semantic-config", new()
{
TitleField = new(){ FieldName = "HotelName" },
ContentFields =
{
new() { FieldName = "Description" }
},
KeywordFields =
{
new() { FieldName = "Category" }
}
})
{
new SemanticConfiguration("my-semantic-config", new()
{
TitleField = new SemanticField("HotelName"),
ContentFields =
{
new SemanticField("Description")
},
KeywordsFields =
{
new SemanticField("Category")
}
})
}
}
};
Expand All @@ -88,9 +78,9 @@ public class Hotel
public string HotelId { get; set; }
public string HotelName { get; set; }
public string Description { get; set; }
public IReadOnlyList<float> DescriptionVector { get; set; }
public ReadOnlyMemory<float> DescriptionVector { get; set; }
public string Category { get; set; }
public IReadOnlyList<float> CategoryVector { get; set; }
public ReadOnlyMemory<float> CategoryVector { get; set; }
}
```

Expand Down Expand Up @@ -154,34 +144,43 @@ await searchClient.IndexDocumentsAsync(IndexDocumentsBatch.Upload(hotelDocuments

## Query Vector Data

When using `RawVectorQuery`, the query for a vector field must also be a vector. To convert a text query string provided by a user into a vector representation, your application must call an embedding library that provides this capability. Use the same embedding library that you used to generate embeddings in the source documents. For more details on how to generate embeddings, please refer to the [documentation](https://learn.microsoft.com/azure/search/vector-search-how-to-generate-embeddings). In the sample codes below, we are using hardcoded embeddings to query vector field.
When using `VectorizedQuery`, the query for a vector field must also be a vector. To convert a text query string provided by a user into a vector representation, your application must call an embedding library that provides this capability. Use the same embedding library that you used to generate embeddings in the source documents. For more details on how to generate embeddings, please refer to the [documentation](https://learn.microsoft.com/azure/search/vector-search-how-to-generate-embeddings). In the sample codes below, we are using hardcoded embeddings to query vector field.

Let's query the index and make sure everything works as implemented. You can also refer to the [documentation](https://learn.microsoft.com/azure/search/vector-search-how-to-query?tabs=portal-vector-query#query-syntax-for-hybrid-search) for more information on querying vector data.

### Vector Semantic Hybrid Query

In a vector semantic hybrid query, the `VectorQueries` collection contains the vectors representing the query input. The `Fields` property specifies which vector fields to search within. The `KNearestNeighborsCount` property dictates the number of nearest neighbors to return as top hits. With the semantic configuration added, we can proceed to execute a semantic hybrid query.
In the context of vector search, the `Queries` collection contains the vectors that represent the query input. The `Fields` property specifies which vector fields should be searched. The `KNearestNeighborsCount` property determines the number of nearest neighbors to retrieve as the top hits.

For semantic search, we will specify `SemanticSearch.SemanticConfigurationName` as `SearchQueryType.Semantic` in the `SearchOptions`. We will use the same `SemanticConfigurationName` that we defined when creating the index. Additionally, we have enabled `SemanticSearch.QueryCaption` and `SemanticSearch.QueryAnswer` in the `SearchOptions` to obtain the caption and answer in the response. With these configurations in place, we are prepared to execute a vector semantic hybrid query.

With these settings in place, we're ready to execute a vector semantic hybrid query:

```C# Snippet:Azure_Search_Documents_Tests_Samples_Sample07_Vector_Semantic_Hybrid_Search
IReadOnlyList<float> vectorizedResult = VectorSearchEmbeddings.SearchVectorizeDescription; // "Top hotels in town"
ReadOnlyMemory<float> vectorizedResult = VectorSearchEmbeddings.SearchVectorizeDescription; // "Top hotels in town"

SearchResults<Hotel> response = await searchClient.SearchAsync<Hotel>(
"Is there any hotel located on the main commercial artery of the city in the heart of New York?",
new SearchOptions
{
VectorQueries = { new RawVectorQuery() { Vector = vectorizedResult, KNearestNeighborsCount = 3, Fields = { "DescriptionVector" } } },
QueryType = SearchQueryType.Semantic,
QueryLanguage = QueryLanguage.EnUs,
SemanticConfigurationName = "my-semantic-config",
QueryCaption = QueryCaptionType.Extractive,
QueryAnswer = QueryAnswerType.Extractive,
});
new SearchOptions
{
VectorSearch = new()
{
Queries = { new VectorizedQuery(vectorizedResult) { KNearestNeighborsCount = 3, Fields = { "DescriptionVector" } } }
},
SemanticSearch = new()
{
SemanticConfigurationName = "my-semantic-config",
QueryCaption = new(QueryCaptionType.Extractive),
QueryAnswer = new(QueryAnswerType.Extractive)
},
QueryType = SearchQueryType.Semantic,
});

int count = 0;
Console.WriteLine($"Semantic Hybrid Search Results:");

Console.WriteLine($"\nQuery Answer:");
foreach (AnswerResult result in response.Answers)
foreach (QueryAnswerResult result in response.SemanticSearch.Answers)
{
Console.WriteLine($"Answer Highlights: {result.Highlights}");
Console.WriteLine($"Answer Text: {result.Text}");
Expand All @@ -193,9 +192,9 @@ await foreach (SearchResult<Hotel> result in response.GetResultsAsync())
Hotel doc = result.Document;
Console.WriteLine($"{doc.HotelId}: {doc.HotelName}");

if (result.Captions != null)
if (result.SemanticSearch.Captions != null)
{
var caption = result.Captions.FirstOrDefault();
var caption = result.SemanticSearch.Captions.FirstOrDefault();
if (caption.Highlights != null && caption.Highlights != "")
{
Console.WriteLine($"Caption Highlights: {caption.Highlights}");
Expand Down
Loading