feat: Add Official Microsoft & Gemini Skills (845+ Total)
🚀 Impact Significantly expands the capabilities of **Antigravity Awesome Skills** by integrating official skill collections from **Microsoft** and **Google Gemini**. This update increases the total skill count to **845+**, making the library even more comprehensive for AI coding assistants. ✨ Key Changes 1. New Official Skills - **Microsoft Skills**: Added a massive collection of official skills from [microsoft/skills](https://github.com/microsoft/skills). - Includes Azure, .NET, Python, TypeScript, and Semantic Kernel skills. - Preserves the original directory structure under `skills/official/microsoft/`. - Includes plugin skills from the `.github/plugins` directory. - **Gemini Skills**: Added official Gemini API development skills under `skills/gemini-api-dev/`. 2. New Scripts & Tooling - **`scripts/sync_microsoft_skills.py`**: A robust synchronization script that: - Clones the official Microsoft repository. - Preserves the original directory heirarchy. - Handles symlinks and plugin locations. - Generates attribution metadata. - **`scripts/tests/inspect_microsoft_repo.py`**: Debug tool to inspect the remote repository structure. - **`scripts/tests/test_comprehensive_coverage.py`**: Verification script to ensure 100% of skills are captured during sync. 3. Core Improvements - **`scripts/generate_index.py`**: Enhanced frontmatter parsing to safely handle unquoted values containing `@` symbols and commas (fixing issues with some Microsoft skill descriptions). - **`package.json`**: Added `sync:microsoft` and `sync:all-official` scripts for easy maintenance. 4. Documentation - Updated `README.md` to reflect the new skill counts (845+) and added Microsoft/Gemini to the provider list. - Updated `CATALOG.md` and `skills_index.json` with the new skills. 🧪 Verification - Ran `scripts/tests/test_comprehensive_coverage.py` to verify all Microsoft skills are detected. - Validated `generate_index.py` fixes by successfully indexing the new skills.
This commit is contained in:
Ahmed Rehan
@@ -0,0 +1,337 @@
|
||||
---
|
||||
name: azure-ai-document-intelligence-dotnet
|
||||
description: |
|
||||
Azure AI Document Intelligence SDK for .NET. Extract text, tables, and structured data from documents using prebuilt and custom models. Use for invoice processing, receipt extraction, ID document analysis, and custom document models. Triggers: "Document Intelligence", "DocumentIntelligenceClient", "form recognizer", "invoice extraction", "receipt OCR", "document analysis .NET".
|
||||
package: Azure.AI.DocumentIntelligence
|
||||
---
|
||||
|
||||
# Azure.AI.DocumentIntelligence (.NET)
|
||||
|
||||
Extract text, tables, and structured data from documents using prebuilt and custom models.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
dotnet add package Azure.AI.DocumentIntelligence
|
||||
dotnet add package Azure.Identity
|
||||
```
|
||||
|
||||
**Current Version**: v1.0.0 (GA)
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
DOCUMENT_INTELLIGENCE_ENDPOINT=https://<resource-name>.cognitiveservices.azure.com/
|
||||
DOCUMENT_INTELLIGENCE_API_KEY=<your-api-key>
|
||||
BLOB_CONTAINER_SAS_URL=https://<storage>.blob.core.windows.net/<container>?<sas-token>
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
### Microsoft Entra ID (Recommended)
|
||||
|
||||
```csharp
|
||||
using Azure.Identity;
|
||||
using Azure.AI.DocumentIntelligence;
|
||||
|
||||
string endpoint = Environment.GetEnvironmentVariable("DOCUMENT_INTELLIGENCE_ENDPOINT");
|
||||
var credential = new DefaultAzureCredential();
|
||||
var client = new DocumentIntelligenceClient(new Uri(endpoint), credential);
|
||||
```
|
||||
|
||||
> **Note**: Entra ID requires a **custom subdomain** (e.g., `https://<resource-name>.cognitiveservices.azure.com/`), not a regional endpoint.
|
||||
|
||||
### API Key
|
||||
|
||||
```csharp
|
||||
string endpoint = Environment.GetEnvironmentVariable("DOCUMENT_INTELLIGENCE_ENDPOINT");
|
||||
string apiKey = Environment.GetEnvironmentVariable("DOCUMENT_INTELLIGENCE_API_KEY");
|
||||
var client = new DocumentIntelligenceClient(new Uri(endpoint), new AzureKeyCredential(apiKey));
|
||||
```
|
||||
|
||||
## Client Types
|
||||
|
||||
| Client | Purpose |
|
||||
|--------|---------|
|
||||
| `DocumentIntelligenceClient` | Analyze documents, classify documents |
|
||||
| `DocumentIntelligenceAdministrationClient` | Build/manage custom models and classifiers |
|
||||
|
||||
## Prebuilt Models
|
||||
|
||||
| Model ID | Description |
|
||||
|----------|-------------|
|
||||
| `prebuilt-read` | Extract text, languages, handwriting |
|
||||
| `prebuilt-layout` | Extract text, tables, selection marks, structure |
|
||||
| `prebuilt-invoice` | Extract invoice fields (vendor, items, totals) |
|
||||
| `prebuilt-receipt` | Extract receipt fields (merchant, items, total) |
|
||||
| `prebuilt-idDocument` | Extract ID document fields (name, DOB, address) |
|
||||
| `prebuilt-businessCard` | Extract business card fields |
|
||||
| `prebuilt-tax.us.w2` | Extract W-2 tax form fields |
|
||||
| `prebuilt-healthInsuranceCard.us` | Extract health insurance card fields |
|
||||
|
||||
## Core Workflows
|
||||
|
||||
### 1. Analyze Invoice
|
||||
|
||||
```csharp
|
||||
using Azure.AI.DocumentIntelligence;
|
||||
|
||||
Uri invoiceUri = new Uri("https://example.com/invoice.pdf");
|
||||
|
||||
Operation<AnalyzeResult> operation = await client.AnalyzeDocumentAsync(
|
||||
WaitUntil.Completed,
|
||||
"prebuilt-invoice",
|
||||
invoiceUri);
|
||||
|
||||
AnalyzeResult result = operation.Value;
|
||||
|
||||
foreach (AnalyzedDocument document in result.Documents)
|
||||
{
|
||||
if (document.Fields.TryGetValue("VendorName", out DocumentField vendorNameField)
|
||||
&& vendorNameField.FieldType == DocumentFieldType.String)
|
||||
{
|
||||
string vendorName = vendorNameField.ValueString;
|
||||
Console.WriteLine($"Vendor Name: '{vendorName}', confidence: {vendorNameField.Confidence}");
|
||||
}
|
||||
|
||||
if (document.Fields.TryGetValue("InvoiceTotal", out DocumentField invoiceTotalField)
|
||||
&& invoiceTotalField.FieldType == DocumentFieldType.Currency)
|
||||
{
|
||||
CurrencyValue invoiceTotal = invoiceTotalField.ValueCurrency;
|
||||
Console.WriteLine($"Invoice Total: '{invoiceTotal.CurrencySymbol}{invoiceTotal.Amount}'");
|
||||
}
|
||||
|
||||
// Extract line items
|
||||
if (document.Fields.TryGetValue("Items", out DocumentField itemsField)
|
||||
&& itemsField.FieldType == DocumentFieldType.List)
|
||||
{
|
||||
foreach (DocumentField item in itemsField.ValueList)
|
||||
{
|
||||
var itemFields = item.ValueDictionary;
|
||||
if (itemFields.TryGetValue("Description", out DocumentField descField))
|
||||
Console.WriteLine($" Item: {descField.ValueString}");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Extract Layout (Text, Tables, Structure)
|
||||
|
||||
```csharp
|
||||
Uri fileUri = new Uri("https://example.com/document.pdf");
|
||||
|
||||
Operation<AnalyzeResult> operation = await client.AnalyzeDocumentAsync(
|
||||
WaitUntil.Completed,
|
||||
"prebuilt-layout",
|
||||
fileUri);
|
||||
|
||||
AnalyzeResult result = operation.Value;
|
||||
|
||||
// Extract text by page
|
||||
foreach (DocumentPage page in result.Pages)
|
||||
{
|
||||
Console.WriteLine($"Page {page.PageNumber}: {page.Lines.Count} lines, {page.Words.Count} words");
|
||||
|
||||
foreach (DocumentLine line in page.Lines)
|
||||
{
|
||||
Console.WriteLine($" Line: '{line.Content}'");
|
||||
}
|
||||
}
|
||||
|
||||
// Extract tables
|
||||
foreach (DocumentTable table in result.Tables)
|
||||
{
|
||||
Console.WriteLine($"Table: {table.RowCount} rows x {table.ColumnCount} columns");
|
||||
foreach (DocumentTableCell cell in table.Cells)
|
||||
{
|
||||
Console.WriteLine($" Cell ({cell.RowIndex}, {cell.ColumnIndex}): {cell.Content}");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Analyze Receipt
|
||||
|
||||
```csharp
|
||||
Operation<AnalyzeResult> operation = await client.AnalyzeDocumentAsync(
|
||||
WaitUntil.Completed,
|
||||
"prebuilt-receipt",
|
||||
receiptUri);
|
||||
|
||||
AnalyzeResult result = operation.Value;
|
||||
|
||||
foreach (AnalyzedDocument document in result.Documents)
|
||||
{
|
||||
if (document.Fields.TryGetValue("MerchantName", out DocumentField merchantField))
|
||||
Console.WriteLine($"Merchant: {merchantField.ValueString}");
|
||||
|
||||
if (document.Fields.TryGetValue("Total", out DocumentField totalField))
|
||||
Console.WriteLine($"Total: {totalField.ValueCurrency.Amount}");
|
||||
|
||||
if (document.Fields.TryGetValue("TransactionDate", out DocumentField dateField))
|
||||
Console.WriteLine($"Date: {dateField.ValueDate}");
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Build Custom Model
|
||||
|
||||
```csharp
|
||||
var adminClient = new DocumentIntelligenceAdministrationClient(
|
||||
new Uri(endpoint),
|
||||
new AzureKeyCredential(apiKey));
|
||||
|
||||
string modelId = "my-custom-model";
|
||||
Uri blobContainerUri = new Uri("<blob-container-sas-url>");
|
||||
|
||||
var blobSource = new BlobContentSource(blobContainerUri);
|
||||
var options = new BuildDocumentModelOptions(modelId, DocumentBuildMode.Template, blobSource);
|
||||
|
||||
Operation<DocumentModelDetails> operation = await adminClient.BuildDocumentModelAsync(
|
||||
WaitUntil.Completed,
|
||||
options);
|
||||
|
||||
DocumentModelDetails model = operation.Value;
|
||||
|
||||
Console.WriteLine($"Model ID: {model.ModelId}");
|
||||
Console.WriteLine($"Created: {model.CreatedOn}");
|
||||
|
||||
foreach (var docType in model.DocumentTypes)
|
||||
{
|
||||
Console.WriteLine($"Document type: {docType.Key}");
|
||||
foreach (var field in docType.Value.FieldSchema)
|
||||
{
|
||||
Console.WriteLine($" Field: {field.Key}, Confidence: {docType.Value.FieldConfidence[field.Key]}");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Build Document Classifier
|
||||
|
||||
```csharp
|
||||
string classifierId = "my-classifier";
|
||||
Uri blobContainerUri = new Uri("<blob-container-sas-url>");
|
||||
|
||||
var sourceA = new BlobContentSource(blobContainerUri) { Prefix = "TypeA/train" };
|
||||
var sourceB = new BlobContentSource(blobContainerUri) { Prefix = "TypeB/train" };
|
||||
|
||||
var docTypes = new Dictionary<string, ClassifierDocumentTypeDetails>()
|
||||
{
|
||||
{ "TypeA", new ClassifierDocumentTypeDetails(sourceA) },
|
||||
{ "TypeB", new ClassifierDocumentTypeDetails(sourceB) }
|
||||
};
|
||||
|
||||
var options = new BuildClassifierOptions(classifierId, docTypes);
|
||||
|
||||
Operation<DocumentClassifierDetails> operation = await adminClient.BuildClassifierAsync(
|
||||
WaitUntil.Completed,
|
||||
options);
|
||||
|
||||
DocumentClassifierDetails classifier = operation.Value;
|
||||
Console.WriteLine($"Classifier ID: {classifier.ClassifierId}");
|
||||
```
|
||||
|
||||
### 6. Classify Document
|
||||
|
||||
```csharp
|
||||
string classifierId = "my-classifier";
|
||||
Uri documentUri = new Uri("https://example.com/document.pdf");
|
||||
|
||||
var options = new ClassifyDocumentOptions(classifierId, documentUri);
|
||||
|
||||
Operation<AnalyzeResult> operation = await client.ClassifyDocumentAsync(
|
||||
WaitUntil.Completed,
|
||||
options);
|
||||
|
||||
AnalyzeResult result = operation.Value;
|
||||
|
||||
foreach (AnalyzedDocument document in result.Documents)
|
||||
{
|
||||
Console.WriteLine($"Document type: {document.DocumentType}, confidence: {document.Confidence}");
|
||||
}
|
||||
```
|
||||
|
||||
### 7. Manage Models
|
||||
|
||||
```csharp
|
||||
// Get resource details
|
||||
DocumentIntelligenceResourceDetails resourceDetails = await adminClient.GetResourceDetailsAsync();
|
||||
Console.WriteLine($"Custom models: {resourceDetails.CustomDocumentModels.Count}/{resourceDetails.CustomDocumentModels.Limit}");
|
||||
|
||||
// Get specific model
|
||||
DocumentModelDetails model = await adminClient.GetModelAsync("my-model-id");
|
||||
Console.WriteLine($"Model: {model.ModelId}, Created: {model.CreatedOn}");
|
||||
|
||||
// List models
|
||||
await foreach (DocumentModelDetails modelItem in adminClient.GetModelsAsync())
|
||||
{
|
||||
Console.WriteLine($"Model: {modelItem.ModelId}");
|
||||
}
|
||||
|
||||
// Delete model
|
||||
await adminClient.DeleteModelAsync("my-model-id");
|
||||
```
|
||||
|
||||
## Key Types Reference
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `DocumentIntelligenceClient` | Main client for analysis |
|
||||
| `DocumentIntelligenceAdministrationClient` | Model management |
|
||||
| `AnalyzeResult` | Result of document analysis |
|
||||
| `AnalyzedDocument` | Single document within result |
|
||||
| `DocumentField` | Extracted field with value and confidence |
|
||||
| `DocumentFieldType` | String, Date, Number, Currency, etc. |
|
||||
| `DocumentPage` | Page info (lines, words, selection marks) |
|
||||
| `DocumentTable` | Extracted table with cells |
|
||||
| `DocumentModelDetails` | Custom model metadata |
|
||||
| `BlobContentSource` | Training data source |
|
||||
|
||||
## Build Modes
|
||||
|
||||
| Mode | Use Case |
|
||||
|------|----------|
|
||||
| `DocumentBuildMode.Template` | Fixed layout documents (forms) |
|
||||
| `DocumentBuildMode.Neural` | Variable layout documents |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use DefaultAzureCredential** for production
|
||||
2. **Reuse client instances** — clients are thread-safe
|
||||
3. **Handle long-running operations** — Use `WaitUntil.Completed` for simplicity
|
||||
4. **Check field confidence** — Always verify `Confidence` property
|
||||
5. **Use appropriate model** — Prebuilt for common docs, custom for specialized
|
||||
6. **Use custom subdomain** — Required for Entra ID authentication
|
||||
|
||||
## Error Handling
|
||||
|
||||
```csharp
|
||||
using Azure;
|
||||
|
||||
try
|
||||
{
|
||||
var operation = await client.AnalyzeDocumentAsync(
|
||||
WaitUntil.Completed,
|
||||
"prebuilt-invoice",
|
||||
documentUri);
|
||||
}
|
||||
catch (RequestFailedException ex)
|
||||
{
|
||||
Console.WriteLine($"Error: {ex.Status} - {ex.Message}");
|
||||
}
|
||||
```
|
||||
|
||||
## Related SDKs
|
||||
|
||||
| SDK | Purpose | Install |
|
||||
|-----|---------|---------|
|
||||
| `Azure.AI.DocumentIntelligence` | Document analysis (this SDK) | `dotnet add package Azure.AI.DocumentIntelligence` |
|
||||
| `Azure.AI.FormRecognizer` | Legacy SDK (deprecated) | Use DocumentIntelligence instead |
|
||||
|
||||
## Reference Links
|
||||
|
||||
| Resource | URL |
|
||||
|----------|-----|
|
||||
| NuGet Package | https://www.nuget.org/packages/Azure.AI.DocumentIntelligence |
|
||||
| API Reference | https://learn.microsoft.com/dotnet/api/azure.ai.documentintelligence |
|
||||
| GitHub Samples | https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/documentintelligence/Azure.AI.DocumentIntelligence/samples |
|
||||
| Document Intelligence Studio | https://documentintelligence.ai.azure.com/ |
|
||||
| Prebuilt Models | https://aka.ms/azsdk/formrecognizer/models |
|
||||
455
skills/official/microsoft/dotnet/foundry/openai/SKILL.md
Normal file
455
skills/official/microsoft/dotnet/foundry/openai/SKILL.md
Normal file
@@ -0,0 +1,455 @@
|
||||
---
|
||||
name: azure-ai-openai-dotnet
|
||||
description: |
|
||||
Azure OpenAI SDK for .NET. Client library for Azure OpenAI and OpenAI services. Use for chat completions, embeddings, image generation, audio transcription, and assistants. Triggers: "Azure OpenAI", "AzureOpenAIClient", "ChatClient", "chat completions .NET", "GPT-4", "embeddings", "DALL-E", "Whisper", "OpenAI .NET".
|
||||
package: Azure.AI.OpenAI
|
||||
---
|
||||
|
||||
# Azure.AI.OpenAI (.NET)
|
||||
|
||||
Client library for Azure OpenAI Service providing access to OpenAI models including GPT-4, GPT-4o, embeddings, DALL-E, and Whisper.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
dotnet add package Azure.AI.OpenAI
|
||||
|
||||
# For OpenAI (non-Azure) compatibility
|
||||
dotnet add package OpenAI
|
||||
```
|
||||
|
||||
**Current Version**: 2.1.0 (stable)
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
AZURE_OPENAI_ENDPOINT=https://<resource-name>.openai.azure.com
|
||||
AZURE_OPENAI_API_KEY=<api-key> # For key-based auth
|
||||
AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o-mini # Your deployment name
|
||||
```
|
||||
|
||||
## Client Hierarchy
|
||||
|
||||
```
|
||||
AzureOpenAIClient (top-level)
|
||||
├── GetChatClient(deploymentName) → ChatClient
|
||||
├── GetEmbeddingClient(deploymentName) → EmbeddingClient
|
||||
├── GetImageClient(deploymentName) → ImageClient
|
||||
├── GetAudioClient(deploymentName) → AudioClient
|
||||
└── GetAssistantClient() → AssistantClient
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
### API Key Authentication
|
||||
|
||||
```csharp
|
||||
using Azure;
|
||||
using Azure.AI.OpenAI;
|
||||
|
||||
AzureOpenAIClient client = new(
|
||||
new Uri(Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT")!),
|
||||
new AzureKeyCredential(Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY")!));
|
||||
```
|
||||
|
||||
### Microsoft Entra ID (Recommended for Production)
|
||||
|
||||
```csharp
|
||||
using Azure.Identity;
|
||||
using Azure.AI.OpenAI;
|
||||
|
||||
AzureOpenAIClient client = new(
|
||||
new Uri(Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT")!),
|
||||
new DefaultAzureCredential());
|
||||
```
|
||||
|
||||
### Using OpenAI SDK Directly with Azure
|
||||
|
||||
```csharp
|
||||
using Azure.Identity;
|
||||
using OpenAI;
|
||||
using OpenAI.Chat;
|
||||
using System.ClientModel.Primitives;
|
||||
|
||||
#pragma warning disable OPENAI001
|
||||
|
||||
BearerTokenPolicy tokenPolicy = new(
|
||||
new DefaultAzureCredential(),
|
||||
"https://cognitiveservices.azure.com/.default");
|
||||
|
||||
ChatClient client = new(
|
||||
model: "gpt-4o-mini",
|
||||
authenticationPolicy: tokenPolicy,
|
||||
options: new OpenAIClientOptions()
|
||||
{
|
||||
Endpoint = new Uri("https://YOUR-RESOURCE.openai.azure.com/openai/v1")
|
||||
});
|
||||
```
|
||||
|
||||
## Chat Completions
|
||||
|
||||
### Basic Chat
|
||||
|
||||
```csharp
|
||||
using Azure.AI.OpenAI;
|
||||
using OpenAI.Chat;
|
||||
|
||||
AzureOpenAIClient azureClient = new(
|
||||
new Uri(endpoint),
|
||||
new DefaultAzureCredential());
|
||||
|
||||
ChatClient chatClient = azureClient.GetChatClient("gpt-4o-mini");
|
||||
|
||||
ChatCompletion completion = chatClient.CompleteChat(
|
||||
[
|
||||
new SystemChatMessage("You are a helpful assistant."),
|
||||
new UserChatMessage("What is Azure OpenAI?")
|
||||
]);
|
||||
|
||||
Console.WriteLine(completion.Content[0].Text);
|
||||
```
|
||||
|
||||
### Async Chat
|
||||
|
||||
```csharp
|
||||
ChatCompletion completion = await chatClient.CompleteChatAsync(
|
||||
[
|
||||
new SystemChatMessage("You are a helpful assistant."),
|
||||
new UserChatMessage("Explain cloud computing in simple terms.")
|
||||
]);
|
||||
|
||||
Console.WriteLine($"Response: {completion.Content[0].Text}");
|
||||
Console.WriteLine($"Tokens used: {completion.Usage.TotalTokenCount}");
|
||||
```
|
||||
|
||||
### Streaming Chat
|
||||
|
||||
```csharp
|
||||
await foreach (StreamingChatCompletionUpdate update
|
||||
in chatClient.CompleteChatStreamingAsync(messages))
|
||||
{
|
||||
if (update.ContentUpdate.Count > 0)
|
||||
{
|
||||
Console.Write(update.ContentUpdate[0].Text);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Chat with Options
|
||||
|
||||
```csharp
|
||||
ChatCompletionOptions options = new()
|
||||
{
|
||||
MaxOutputTokenCount = 1000,
|
||||
Temperature = 0.7f,
|
||||
TopP = 0.95f,
|
||||
FrequencyPenalty = 0,
|
||||
PresencePenalty = 0
|
||||
};
|
||||
|
||||
ChatCompletion completion = await chatClient.CompleteChatAsync(messages, options);
|
||||
```
|
||||
|
||||
### Multi-turn Conversation
|
||||
|
||||
```csharp
|
||||
List<ChatMessage> messages = new()
|
||||
{
|
||||
new SystemChatMessage("You are a helpful assistant."),
|
||||
new UserChatMessage("Hi, can you help me?"),
|
||||
new AssistantChatMessage("Of course! What do you need help with?"),
|
||||
new UserChatMessage("What's the capital of France?")
|
||||
};
|
||||
|
||||
ChatCompletion completion = await chatClient.CompleteChatAsync(messages);
|
||||
messages.Add(new AssistantChatMessage(completion.Content[0].Text));
|
||||
```
|
||||
|
||||
## Structured Outputs (JSON Schema)
|
||||
|
||||
```csharp
|
||||
using System.Text.Json;
|
||||
|
||||
ChatCompletionOptions options = new()
|
||||
{
|
||||
ResponseFormat = ChatResponseFormat.CreateJsonSchemaFormat(
|
||||
jsonSchemaFormatName: "math_reasoning",
|
||||
jsonSchema: BinaryData.FromBytes("""
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"steps": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"explanation": { "type": "string" },
|
||||
"output": { "type": "string" }
|
||||
},
|
||||
"required": ["explanation", "output"],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"final_answer": { "type": "string" }
|
||||
},
|
||||
"required": ["steps", "final_answer"],
|
||||
"additionalProperties": false
|
||||
}
|
||||
"""u8.ToArray()),
|
||||
jsonSchemaIsStrict: true)
|
||||
};
|
||||
|
||||
ChatCompletion completion = await chatClient.CompleteChatAsync(
|
||||
[new UserChatMessage("How can I solve 8x + 7 = -23?")],
|
||||
options);
|
||||
|
||||
using JsonDocument json = JsonDocument.Parse(completion.Content[0].Text);
|
||||
Console.WriteLine($"Answer: {json.RootElement.GetProperty("final_answer")}");
|
||||
```
|
||||
|
||||
## Reasoning Models (o1, o4-mini)
|
||||
|
||||
```csharp
|
||||
ChatCompletionOptions options = new()
|
||||
{
|
||||
ReasoningEffortLevel = ChatReasoningEffortLevel.Low,
|
||||
MaxOutputTokenCount = 100000
|
||||
};
|
||||
|
||||
ChatCompletion completion = await chatClient.CompleteChatAsync(
|
||||
[
|
||||
new DeveloperChatMessage("You are a helpful assistant"),
|
||||
new UserChatMessage("Explain the theory of relativity")
|
||||
], options);
|
||||
```
|
||||
|
||||
## Azure AI Search Integration (RAG)
|
||||
|
||||
```csharp
|
||||
using Azure.AI.OpenAI.Chat;
|
||||
|
||||
#pragma warning disable AOAI001
|
||||
|
||||
ChatCompletionOptions options = new();
|
||||
options.AddDataSource(new AzureSearchChatDataSource()
|
||||
{
|
||||
Endpoint = new Uri(searchEndpoint),
|
||||
IndexName = searchIndex,
|
||||
Authentication = DataSourceAuthentication.FromApiKey(searchKey)
|
||||
});
|
||||
|
||||
ChatCompletion completion = await chatClient.CompleteChatAsync(
|
||||
[new UserChatMessage("What health plans are available?")],
|
||||
options);
|
||||
|
||||
ChatMessageContext context = completion.GetMessageContext();
|
||||
if (context?.Intent is not null)
|
||||
{
|
||||
Console.WriteLine($"Intent: {context.Intent}");
|
||||
}
|
||||
foreach (ChatCitation citation in context?.Citations ?? [])
|
||||
{
|
||||
Console.WriteLine($"Citation: {citation.Content}");
|
||||
}
|
||||
```
|
||||
|
||||
## Embeddings
|
||||
|
||||
```csharp
|
||||
using OpenAI.Embeddings;
|
||||
|
||||
EmbeddingClient embeddingClient = azureClient.GetEmbeddingClient("text-embedding-ada-002");
|
||||
|
||||
OpenAIEmbedding embedding = await embeddingClient.GenerateEmbeddingAsync("Hello, world!");
|
||||
ReadOnlyMemory<float> vector = embedding.ToFloats();
|
||||
|
||||
Console.WriteLine($"Embedding dimensions: {vector.Length}");
|
||||
```
|
||||
|
||||
### Batch Embeddings
|
||||
|
||||
```csharp
|
||||
List<string> inputs = new()
|
||||
{
|
||||
"First document text",
|
||||
"Second document text",
|
||||
"Third document text"
|
||||
};
|
||||
|
||||
OpenAIEmbeddingCollection embeddings = await embeddingClient.GenerateEmbeddingsAsync(inputs);
|
||||
|
||||
foreach (OpenAIEmbedding emb in embeddings)
|
||||
{
|
||||
Console.WriteLine($"Index {emb.Index}: {emb.ToFloats().Length} dimensions");
|
||||
}
|
||||
```
|
||||
|
||||
## Image Generation (DALL-E)
|
||||
|
||||
```csharp
|
||||
using OpenAI.Images;
|
||||
|
||||
ImageClient imageClient = azureClient.GetImageClient("dall-e-3");
|
||||
|
||||
GeneratedImage image = await imageClient.GenerateImageAsync(
|
||||
"A futuristic city skyline at sunset",
|
||||
new ImageGenerationOptions
|
||||
{
|
||||
Size = GeneratedImageSize.W1024xH1024,
|
||||
Quality = GeneratedImageQuality.High,
|
||||
Style = GeneratedImageStyle.Vivid
|
||||
});
|
||||
|
||||
Console.WriteLine($"Image URL: {image.ImageUri}");
|
||||
```
|
||||
|
||||
## Audio (Whisper)
|
||||
|
||||
### Transcription
|
||||
|
||||
```csharp
|
||||
using OpenAI.Audio;
|
||||
|
||||
AudioClient audioClient = azureClient.GetAudioClient("whisper");
|
||||
|
||||
AudioTranscription transcription = await audioClient.TranscribeAudioAsync(
|
||||
"audio.mp3",
|
||||
new AudioTranscriptionOptions
|
||||
{
|
||||
ResponseFormat = AudioTranscriptionFormat.Verbose,
|
||||
Language = "en"
|
||||
});
|
||||
|
||||
Console.WriteLine(transcription.Text);
|
||||
```
|
||||
|
||||
### Text-to-Speech
|
||||
|
||||
```csharp
|
||||
BinaryData speech = await audioClient.GenerateSpeechAsync(
|
||||
"Hello, welcome to Azure OpenAI!",
|
||||
GeneratedSpeechVoice.Alloy,
|
||||
new SpeechGenerationOptions
|
||||
{
|
||||
SpeedRatio = 1.0f,
|
||||
ResponseFormat = GeneratedSpeechFormat.Mp3
|
||||
});
|
||||
|
||||
await File.WriteAllBytesAsync("output.mp3", speech.ToArray());
|
||||
```
|
||||
|
||||
## Function Calling (Tools)
|
||||
|
||||
```csharp
|
||||
ChatTool getCurrentWeatherTool = ChatTool.CreateFunctionTool(
|
||||
functionName: "get_current_weather",
|
||||
functionDescription: "Get the current weather in a given location",
|
||||
functionParameters: BinaryData.FromString("""
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state, e.g. San Francisco, CA"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"]
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
"""));
|
||||
|
||||
ChatCompletionOptions options = new()
|
||||
{
|
||||
Tools = { getCurrentWeatherTool }
|
||||
};
|
||||
|
||||
ChatCompletion completion = await chatClient.CompleteChatAsync(
|
||||
[new UserChatMessage("What's the weather in Seattle?")],
|
||||
options);
|
||||
|
||||
if (completion.FinishReason == ChatFinishReason.ToolCalls)
|
||||
{
|
||||
foreach (ChatToolCall toolCall in completion.ToolCalls)
|
||||
{
|
||||
Console.WriteLine($"Function: {toolCall.FunctionName}");
|
||||
Console.WriteLine($"Arguments: {toolCall.FunctionArguments}");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Key Types Reference
|
||||
|
||||
| Type | Purpose |
|
||||
|------|---------|
|
||||
| `AzureOpenAIClient` | Top-level client for Azure OpenAI |
|
||||
| `ChatClient` | Chat completions |
|
||||
| `EmbeddingClient` | Text embeddings |
|
||||
| `ImageClient` | Image generation (DALL-E) |
|
||||
| `AudioClient` | Audio transcription/TTS |
|
||||
| `ChatCompletion` | Chat response |
|
||||
| `ChatCompletionOptions` | Request configuration |
|
||||
| `StreamingChatCompletionUpdate` | Streaming response chunk |
|
||||
| `ChatMessage` | Base message type |
|
||||
| `SystemChatMessage` | System prompt |
|
||||
| `UserChatMessage` | User input |
|
||||
| `AssistantChatMessage` | Assistant response |
|
||||
| `DeveloperChatMessage` | Developer message (reasoning models) |
|
||||
| `ChatTool` | Function/tool definition |
|
||||
| `ChatToolCall` | Tool invocation request |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use Entra ID in production** — Avoid API keys; use `DefaultAzureCredential`
|
||||
2. **Reuse client instances** — Create once, share across requests
|
||||
3. **Handle rate limits** — Implement exponential backoff for 429 errors
|
||||
4. **Stream for long responses** — Use `CompleteChatStreamingAsync` for better UX
|
||||
5. **Set appropriate timeouts** — Long completions may need extended timeouts
|
||||
6. **Use structured outputs** — JSON schema ensures consistent response format
|
||||
7. **Monitor token usage** — Track `completion.Usage` for cost management
|
||||
8. **Validate tool calls** — Always validate function arguments before execution
|
||||
|
||||
## Error Handling
|
||||
|
||||
```csharp
|
||||
using Azure;
|
||||
|
||||
try
|
||||
{
|
||||
ChatCompletion completion = await chatClient.CompleteChatAsync(messages);
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 429)
|
||||
{
|
||||
Console.WriteLine("Rate limited. Retry after delay.");
|
||||
await Task.Delay(TimeSpan.FromSeconds(10));
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 400)
|
||||
{
|
||||
Console.WriteLine($"Bad request: {ex.Message}");
|
||||
}
|
||||
catch (RequestFailedException ex)
|
||||
{
|
||||
Console.WriteLine($"Azure OpenAI error: {ex.Status} - {ex.Message}");
|
||||
}
|
||||
```
|
||||
|
||||
## Related SDKs
|
||||
|
||||
| SDK | Purpose | Install |
|
||||
|-----|---------|---------|
|
||||
| `Azure.AI.OpenAI` | Azure OpenAI client (this SDK) | `dotnet add package Azure.AI.OpenAI` |
|
||||
| `OpenAI` | OpenAI compatibility | `dotnet add package OpenAI` |
|
||||
| `Azure.Identity` | Authentication | `dotnet add package Azure.Identity` |
|
||||
| `Azure.Search.Documents` | AI Search for RAG | `dotnet add package Azure.Search.Documents` |
|
||||
|
||||
## Reference Links
|
||||
|
||||
| Resource | URL |
|
||||
|----------|-----|
|
||||
| NuGet Package | https://www.nuget.org/packages/Azure.AI.OpenAI |
|
||||
| API Reference | https://learn.microsoft.com/dotnet/api/azure.ai.openai |
|
||||
| Migration Guide (1.0→2.0) | https://learn.microsoft.com/azure/ai-services/openai/how-to/dotnet-migration |
|
||||
| Quickstart | https://learn.microsoft.com/azure/ai-services/openai/quickstart |
|
||||
| GitHub Source | https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/openai/Azure.AI.OpenAI |
|
||||
348
skills/official/microsoft/dotnet/foundry/projects/SKILL.md
Normal file
348
skills/official/microsoft/dotnet/foundry/projects/SKILL.md
Normal file
@@ -0,0 +1,348 @@
|
||||
---
|
||||
name: azure-ai-projects-dotnet
|
||||
description: |
|
||||
Azure AI Projects SDK for .NET. High-level client for Azure AI Foundry projects including agents, connections, datasets, deployments, evaluations, and indexes. Use for AI Foundry project management, versioned agents, and orchestration. Triggers: "AI Projects", "AIProjectClient", "Foundry project", "versioned agents", "evaluations", "datasets", "connections", "deployments .NET".
|
||||
package: Azure.AI.Projects
|
||||
---
|
||||
|
||||
# Azure.AI.Projects (.NET)
|
||||
|
||||
High-level SDK for Azure AI Foundry project operations including agents, connections, datasets, deployments, evaluations, and indexes.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
dotnet add package Azure.AI.Projects
|
||||
dotnet add package Azure.Identity
|
||||
|
||||
# Optional: For versioned agents with OpenAI extensions
|
||||
dotnet add package Azure.AI.Projects.OpenAI --prerelease
|
||||
|
||||
# Optional: For low-level agent operations
|
||||
dotnet add package Azure.AI.Agents.Persistent --prerelease
|
||||
```
|
||||
|
||||
**Current Versions**: GA v1.1.0, Preview v1.2.0-beta.5
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
PROJECT_ENDPOINT=https://<resource>.services.ai.azure.com/api/projects/<project>
|
||||
MODEL_DEPLOYMENT_NAME=gpt-4o-mini
|
||||
CONNECTION_NAME=<your-connection-name>
|
||||
AI_SEARCH_CONNECTION_NAME=<ai-search-connection>
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
```csharp
|
||||
using Azure.Identity;
|
||||
using Azure.AI.Projects;
|
||||
|
||||
var endpoint = Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
|
||||
AIProjectClient projectClient = new AIProjectClient(
|
||||
new Uri(endpoint),
|
||||
new DefaultAzureCredential());
|
||||
```
|
||||
|
||||
## Client Hierarchy
|
||||
|
||||
```
|
||||
AIProjectClient
|
||||
├── Agents → AIProjectAgentsOperations (versioned agents)
|
||||
├── Connections → ConnectionsClient
|
||||
├── Datasets → DatasetsClient
|
||||
├── Deployments → DeploymentsClient
|
||||
├── Evaluations → EvaluationsClient
|
||||
├── Evaluators → EvaluatorsClient
|
||||
├── Indexes → IndexesClient
|
||||
├── Telemetry → AIProjectTelemetry
|
||||
├── OpenAI → ProjectOpenAIClient (preview)
|
||||
└── GetPersistentAgentsClient() → PersistentAgentsClient
|
||||
```
|
||||
|
||||
## Core Workflows
|
||||
|
||||
### 1. Get Persistent Agents Client
|
||||
|
||||
```csharp
|
||||
// Get low-level agents client from project client
|
||||
PersistentAgentsClient agentsClient = projectClient.GetPersistentAgentsClient();
|
||||
|
||||
// Create agent
|
||||
PersistentAgent agent = await agentsClient.Administration.CreateAgentAsync(
|
||||
model: "gpt-4o-mini",
|
||||
name: "Math Tutor",
|
||||
instructions: "You are a personal math tutor.");
|
||||
|
||||
// Create thread and run
|
||||
PersistentAgentThread thread = await agentsClient.Threads.CreateThreadAsync();
|
||||
await agentsClient.Messages.CreateMessageAsync(thread.Id, MessageRole.User, "Solve 3x + 11 = 14");
|
||||
ThreadRun run = await agentsClient.Runs.CreateRunAsync(thread.Id, agent.Id);
|
||||
|
||||
// Poll for completion
|
||||
do
|
||||
{
|
||||
await Task.Delay(500);
|
||||
run = await agentsClient.Runs.GetRunAsync(thread.Id, run.Id);
|
||||
}
|
||||
while (run.Status == RunStatus.Queued || run.Status == RunStatus.InProgress);
|
||||
|
||||
// Get messages
|
||||
await foreach (var msg in agentsClient.Messages.GetMessagesAsync(thread.Id))
|
||||
{
|
||||
foreach (var content in msg.ContentItems)
|
||||
{
|
||||
if (content is MessageTextContent textContent)
|
||||
Console.WriteLine(textContent.Text);
|
||||
}
|
||||
}
|
||||
|
||||
// Cleanup
|
||||
await agentsClient.Threads.DeleteThreadAsync(thread.Id);
|
||||
await agentsClient.Administration.DeleteAgentAsync(agent.Id);
|
||||
```
|
||||
|
||||
### 2. Versioned Agents with Tools (Preview)
|
||||
|
||||
```csharp
|
||||
using Azure.AI.Projects.OpenAI;
|
||||
|
||||
// Create agent with web search tool
|
||||
PromptAgentDefinition agentDefinition = new(model: "gpt-4o-mini")
|
||||
{
|
||||
Instructions = "You are a helpful assistant that can search the web",
|
||||
Tools = {
|
||||
ResponseTool.CreateWebSearchTool(
|
||||
userLocation: WebSearchToolLocation.CreateApproximateLocation(
|
||||
country: "US",
|
||||
city: "Seattle",
|
||||
region: "Washington"
|
||||
)
|
||||
),
|
||||
}
|
||||
};
|
||||
|
||||
AgentVersion agentVersion = await projectClient.Agents.CreateAgentVersionAsync(
|
||||
agentName: "myAgent",
|
||||
options: new(agentDefinition));
|
||||
|
||||
// Get response client
|
||||
ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
|
||||
|
||||
// Create response
|
||||
ResponseResult response = responseClient.CreateResponse("What's the weather in Seattle?");
|
||||
Console.WriteLine(response.GetOutputText());
|
||||
|
||||
// Cleanup
|
||||
projectClient.Agents.DeleteAgentVersion(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
|
||||
```
|
||||
|
||||
### 3. Connections
|
||||
|
||||
```csharp
|
||||
// List all connections
|
||||
foreach (AIProjectConnection connection in projectClient.Connections.GetConnections())
|
||||
{
|
||||
Console.WriteLine($"{connection.Name}: {connection.ConnectionType}");
|
||||
}
|
||||
|
||||
// Get specific connection
|
||||
AIProjectConnection conn = projectClient.Connections.GetConnection(
|
||||
connectionName,
|
||||
includeCredentials: true);
|
||||
|
||||
// Get default connection
|
||||
AIProjectConnection defaultConn = projectClient.Connections.GetDefaultConnection(
|
||||
includeCredentials: false);
|
||||
```
|
||||
|
||||
### 4. Deployments
|
||||
|
||||
```csharp
|
||||
// List all deployments
|
||||
foreach (AIProjectDeployment deployment in projectClient.Deployments.GetDeployments())
|
||||
{
|
||||
Console.WriteLine($"{deployment.Name}: {deployment.ModelName}");
|
||||
}
|
||||
|
||||
// Filter by publisher
|
||||
foreach (var deployment in projectClient.Deployments.GetDeployments(modelPublisher: "Microsoft"))
|
||||
{
|
||||
Console.WriteLine(deployment.Name);
|
||||
}
|
||||
|
||||
// Get specific deployment
|
||||
ModelDeployment details = (ModelDeployment)projectClient.Deployments.GetDeployment("gpt-4o-mini");
|
||||
```
|
||||
|
||||
### 5. Datasets
|
||||
|
||||
```csharp
|
||||
// Upload single file
|
||||
FileDataset fileDataset = projectClient.Datasets.UploadFile(
|
||||
name: "my-dataset",
|
||||
version: "1.0",
|
||||
filePath: "data/training.txt",
|
||||
connectionName: connectionName);
|
||||
|
||||
// Upload folder
|
||||
FolderDataset folderDataset = projectClient.Datasets.UploadFolder(
|
||||
name: "my-dataset",
|
||||
version: "2.0",
|
||||
folderPath: "data/training",
|
||||
connectionName: connectionName,
|
||||
filePattern: new Regex(".*\\.txt"));
|
||||
|
||||
// Get dataset
|
||||
AIProjectDataset dataset = projectClient.Datasets.GetDataset("my-dataset", "1.0");
|
||||
|
||||
// Delete dataset
|
||||
projectClient.Datasets.Delete("my-dataset", "1.0");
|
||||
```
|
||||
|
||||
### 6. Indexes
|
||||
|
||||
```csharp
|
||||
// Create Azure AI Search index
|
||||
AzureAISearchIndex searchIndex = new(aiSearchConnectionName, aiSearchIndexName)
|
||||
{
|
||||
Description = "Sample Index"
|
||||
};
|
||||
|
||||
searchIndex = (AzureAISearchIndex)projectClient.Indexes.CreateOrUpdate(
|
||||
name: "my-index",
|
||||
version: "1.0",
|
||||
index: searchIndex);
|
||||
|
||||
// List indexes
|
||||
foreach (AIProjectIndex index in projectClient.Indexes.GetIndexes())
|
||||
{
|
||||
Console.WriteLine(index.Name);
|
||||
}
|
||||
|
||||
// Delete index
|
||||
projectClient.Indexes.Delete(name: "my-index", version: "1.0");
|
||||
```
|
||||
|
||||
### 7. Evaluations
|
||||
|
||||
```csharp
|
||||
// Create evaluation configuration
|
||||
var evaluatorConfig = new EvaluatorConfiguration(id: EvaluatorIDs.Relevance);
|
||||
evaluatorConfig.InitParams.Add("deployment_name", BinaryData.FromObjectAsJson("gpt-4o"));
|
||||
|
||||
// Create evaluation
|
||||
Evaluation evaluation = new Evaluation(
|
||||
data: new InputDataset("<dataset_id>"),
|
||||
evaluators: new Dictionary<string, EvaluatorConfiguration>
|
||||
{
|
||||
{ "relevance", evaluatorConfig }
|
||||
}
|
||||
)
|
||||
{
|
||||
DisplayName = "Sample Evaluation"
|
||||
};
|
||||
|
||||
// Run evaluation
|
||||
Evaluation result = projectClient.Evaluations.Create(evaluation: evaluation);
|
||||
|
||||
// Get evaluation
|
||||
Evaluation getResult = projectClient.Evaluations.Get(result.Name);
|
||||
|
||||
// List evaluations
|
||||
foreach (var eval in projectClient.Evaluations.GetAll())
|
||||
{
|
||||
Console.WriteLine($"{eval.DisplayName}: {eval.Status}");
|
||||
}
|
||||
```
|
||||
|
||||
### 8. Get Azure OpenAI Chat Client
|
||||
|
||||
```csharp
|
||||
using Azure.AI.OpenAI;
|
||||
using OpenAI.Chat;
|
||||
|
||||
ClientConnection connection = projectClient.GetConnection(typeof(AzureOpenAIClient).FullName!);
|
||||
|
||||
if (!connection.TryGetLocatorAsUri(out Uri uri) || uri is null)
|
||||
throw new InvalidOperationException("Invalid URI.");
|
||||
|
||||
uri = new Uri($"https://{uri.Host}");
|
||||
|
||||
AzureOpenAIClient azureOpenAIClient = new AzureOpenAIClient(uri, new DefaultAzureCredential());
|
||||
ChatClient chatClient = azureOpenAIClient.GetChatClient("gpt-4o-mini");
|
||||
|
||||
ChatCompletion result = chatClient.CompleteChat("List all rainbow colors");
|
||||
Console.WriteLine(result.Content[0].Text);
|
||||
```
|
||||
|
||||
## Available Agent Tools
|
||||
|
||||
| Tool | Class | Purpose |
|
||||
|------|-------|---------|
|
||||
| Code Interpreter | `CodeInterpreterToolDefinition` | Execute Python code |
|
||||
| File Search | `FileSearchToolDefinition` | Search uploaded files |
|
||||
| Function Calling | `FunctionToolDefinition` | Call custom functions |
|
||||
| Bing Grounding | `BingGroundingToolDefinition` | Web search via Bing |
|
||||
| Azure AI Search | `AzureAISearchToolDefinition` | Search Azure AI indexes |
|
||||
| OpenAPI | `OpenApiToolDefinition` | Call external APIs |
|
||||
| Azure Functions | `AzureFunctionToolDefinition` | Invoke Azure Functions |
|
||||
| MCP | `MCPToolDefinition` | Model Context Protocol tools |
|
||||
|
||||
## Key Types Reference
|
||||
|
||||
| Type | Purpose |
|
||||
|------|---------|
|
||||
| `AIProjectClient` | Main entry point |
|
||||
| `PersistentAgentsClient` | Low-level agent operations |
|
||||
| `PromptAgentDefinition` | Versioned agent definition |
|
||||
| `AgentVersion` | Versioned agent instance |
|
||||
| `AIProjectConnection` | Connection to Azure resource |
|
||||
| `AIProjectDeployment` | Model deployment info |
|
||||
| `AIProjectDataset` | Dataset metadata |
|
||||
| `AIProjectIndex` | Search index metadata |
|
||||
| `Evaluation` | Evaluation configuration and results |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use `DefaultAzureCredential`** for production authentication
|
||||
2. **Use async methods** (`*Async`) for all I/O operations
|
||||
3. **Poll with appropriate delays** (500ms recommended) when waiting for runs
|
||||
4. **Clean up resources** — delete threads, agents, and files when done
|
||||
5. **Use versioned agents** (via `Azure.AI.Projects.OpenAI`) for production scenarios
|
||||
6. **Store connection IDs** rather than names for tool configurations
|
||||
7. **Use `includeCredentials: true`** only when credentials are needed
|
||||
8. **Handle pagination** — use `AsyncPageable<T>` for listing operations
|
||||
|
||||
## Error Handling
|
||||
|
||||
```csharp
|
||||
using Azure;
|
||||
|
||||
try
|
||||
{
|
||||
var result = await projectClient.Evaluations.CreateAsync(evaluation);
|
||||
}
|
||||
catch (RequestFailedException ex)
|
||||
{
|
||||
Console.WriteLine($"Error: {ex.Status} - {ex.ErrorCode}: {ex.Message}");
|
||||
}
|
||||
```
|
||||
|
||||
## Related SDKs
|
||||
|
||||
| SDK | Purpose | Install |
|
||||
|-----|---------|---------|
|
||||
| `Azure.AI.Projects` | High-level project client (this SDK) | `dotnet add package Azure.AI.Projects` |
|
||||
| `Azure.AI.Agents.Persistent` | Low-level agent operations | `dotnet add package Azure.AI.Agents.Persistent` |
|
||||
| `Azure.AI.Projects.OpenAI` | Versioned agents with OpenAI | `dotnet add package Azure.AI.Projects.OpenAI` |
|
||||
|
||||
## Reference Links
|
||||
|
||||
| Resource | URL |
|
||||
|----------|-----|
|
||||
| NuGet Package | https://www.nuget.org/packages/Azure.AI.Projects |
|
||||
| API Reference | https://learn.microsoft.com/dotnet/api/azure.ai.projects |
|
||||
| GitHub Source | https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/ai/Azure.AI.Projects |
|
||||
| Samples | https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/ai/Azure.AI.Projects/samples |
|
||||
@@ -0,0 +1,339 @@
|
||||
---
|
||||
name: azure-search-documents-dotnet
|
||||
description: |
|
||||
Azure AI Search SDK for .NET (Azure.Search.Documents). Use for building search applications with full-text, vector, semantic, and hybrid search. Covers SearchClient (queries, document CRUD), SearchIndexClient (index management), and SearchIndexerClient (indexers, skillsets). Triggers: "Azure Search .NET", "SearchClient", "SearchIndexClient", "vector search C#", "semantic search .NET", "hybrid search", "Azure.Search.Documents".
|
||||
package: Azure.Search.Documents
|
||||
---
|
||||
|
||||
# Azure.Search.Documents (.NET)
|
||||
|
||||
Build search applications with full-text, vector, semantic, and hybrid search capabilities.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
dotnet add package Azure.Search.Documents
|
||||
dotnet add package Azure.Identity
|
||||
```
|
||||
|
||||
**Current Versions**: Stable v11.7.0, Preview v11.8.0-beta.1
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
SEARCH_ENDPOINT=https://<search-service>.search.windows.net
|
||||
SEARCH_INDEX_NAME=<index-name>
|
||||
# For API key auth (not recommended for production)
|
||||
SEARCH_API_KEY=<api-key>
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
**DefaultAzureCredential (preferred)**:
|
||||
```csharp
|
||||
using Azure.Identity;
|
||||
using Azure.Search.Documents;
|
||||
|
||||
var credential = new DefaultAzureCredential();
|
||||
var client = new SearchClient(
|
||||
new Uri(Environment.GetEnvironmentVariable("SEARCH_ENDPOINT")),
|
||||
Environment.GetEnvironmentVariable("SEARCH_INDEX_NAME"),
|
||||
credential);
|
||||
```
|
||||
|
||||
**API Key**:
|
||||
```csharp
|
||||
using Azure;
|
||||
using Azure.Search.Documents;
|
||||
|
||||
var credential = new AzureKeyCredential(
|
||||
Environment.GetEnvironmentVariable("SEARCH_API_KEY"));
|
||||
var client = new SearchClient(
|
||||
new Uri(Environment.GetEnvironmentVariable("SEARCH_ENDPOINT")),
|
||||
Environment.GetEnvironmentVariable("SEARCH_INDEX_NAME"),
|
||||
credential);
|
||||
```
|
||||
|
||||
## Client Selection
|
||||
|
||||
| Client | Purpose |
|
||||
|--------|---------|
|
||||
| `SearchClient` | Query indexes, upload/update/delete documents |
|
||||
| `SearchIndexClient` | Create/manage indexes, synonym maps |
|
||||
| `SearchIndexerClient` | Manage indexers, skillsets, data sources |
|
||||
|
||||
## Index Creation
|
||||
|
||||
### Using FieldBuilder (Recommended)
|
||||
|
||||
```csharp
|
||||
using Azure.Search.Documents.Indexes;
|
||||
using Azure.Search.Documents.Indexes.Models;
|
||||
|
||||
// Define model with attributes
|
||||
public class Hotel
|
||||
{
|
||||
[SimpleField(IsKey = true, IsFilterable = true)]
|
||||
public string HotelId { get; set; }
|
||||
|
||||
[SearchableField(IsSortable = true)]
|
||||
public string HotelName { get; set; }
|
||||
|
||||
[SearchableField(AnalyzerName = LexicalAnalyzerName.EnLucene)]
|
||||
public string Description { get; set; }
|
||||
|
||||
[SimpleField(IsFilterable = true, IsSortable = true, IsFacetable = true)]
|
||||
public double? Rating { get; set; }
|
||||
|
||||
[VectorSearchField(VectorSearchDimensions = 1536, VectorSearchProfileName = "vector-profile")]
|
||||
public ReadOnlyMemory<float>? DescriptionVector { get; set; }
|
||||
}
|
||||
|
||||
// Create index
|
||||
var indexClient = new SearchIndexClient(endpoint, credential);
|
||||
var fieldBuilder = new FieldBuilder();
|
||||
var fields = fieldBuilder.Build(typeof(Hotel));
|
||||
|
||||
var index = new SearchIndex("hotels")
|
||||
{
|
||||
Fields = fields,
|
||||
VectorSearch = new VectorSearch
|
||||
{
|
||||
Profiles = { new VectorSearchProfile("vector-profile", "hnsw-algo") },
|
||||
Algorithms = { new HnswAlgorithmConfiguration("hnsw-algo") }
|
||||
}
|
||||
};
|
||||
|
||||
await indexClient.CreateOrUpdateIndexAsync(index);
|
||||
```
|
||||
|
||||
### Manual Field Definition
|
||||
|
||||
```csharp
|
||||
var index = new SearchIndex("hotels")
|
||||
{
|
||||
Fields =
|
||||
{
|
||||
new SimpleField("hotelId", SearchFieldDataType.String) { IsKey = true, IsFilterable = true },
|
||||
new SearchableField("hotelName") { IsSortable = true },
|
||||
new SearchableField("description") { AnalyzerName = LexicalAnalyzerName.EnLucene },
|
||||
new SimpleField("rating", SearchFieldDataType.Double) { IsFilterable = true, IsSortable = true },
|
||||
new SearchField("descriptionVector", SearchFieldDataType.Collection(SearchFieldDataType.Single))
|
||||
{
|
||||
VectorSearchDimensions = 1536,
|
||||
VectorSearchProfileName = "vector-profile"
|
||||
}
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Document Operations
|
||||
|
||||
```csharp
|
||||
var searchClient = new SearchClient(endpoint, indexName, credential);
|
||||
|
||||
// Upload (add new)
|
||||
var hotels = new[] { new Hotel { HotelId = "1", HotelName = "Hotel A" } };
|
||||
await searchClient.UploadDocumentsAsync(hotels);
|
||||
|
||||
// Merge (update existing)
|
||||
await searchClient.MergeDocumentsAsync(hotels);
|
||||
|
||||
// Merge or Upload (upsert)
|
||||
await searchClient.MergeOrUploadDocumentsAsync(hotels);
|
||||
|
||||
// Delete
|
||||
await searchClient.DeleteDocumentsAsync("hotelId", new[] { "1", "2" });
|
||||
|
||||
// Batch operations
|
||||
var batch = IndexDocumentsBatch.Create(
|
||||
IndexDocumentsAction.Upload(hotel1),
|
||||
IndexDocumentsAction.Merge(hotel2),
|
||||
IndexDocumentsAction.Delete(hotel3));
|
||||
await searchClient.IndexDocumentsAsync(batch);
|
||||
```
|
||||
|
||||
## Search Patterns
|
||||
|
||||
### Basic Search
|
||||
|
||||
```csharp
|
||||
var options = new SearchOptions
|
||||
{
|
||||
Filter = "rating ge 4",
|
||||
OrderBy = { "rating desc" },
|
||||
Select = { "hotelId", "hotelName", "rating" },
|
||||
Size = 10,
|
||||
Skip = 0,
|
||||
IncludeTotalCount = true
|
||||
};
|
||||
|
||||
SearchResults<Hotel> results = await searchClient.SearchAsync<Hotel>("luxury", options);
|
||||
|
||||
Console.WriteLine($"Total: {results.TotalCount}");
|
||||
await foreach (SearchResult<Hotel> result in results.GetResultsAsync())
|
||||
{
|
||||
Console.WriteLine($"{result.Document.HotelName} (Score: {result.Score})");
|
||||
}
|
||||
```
|
||||
|
||||
### Faceted Search
|
||||
|
||||
```csharp
|
||||
var options = new SearchOptions
|
||||
{
|
||||
Facets = { "rating,count:5", "category" }
|
||||
};
|
||||
|
||||
var results = await searchClient.SearchAsync<Hotel>("*", options);
|
||||
|
||||
foreach (var facet in results.Value.Facets["rating"])
|
||||
{
|
||||
Console.WriteLine($"Rating {facet.Value}: {facet.Count}");
|
||||
}
|
||||
```
|
||||
|
||||
### Autocomplete and Suggestions
|
||||
|
||||
```csharp
|
||||
// Autocomplete
|
||||
var autocompleteOptions = new AutocompleteOptions { Mode = AutocompleteMode.OneTermWithContext };
|
||||
var autocomplete = await searchClient.AutocompleteAsync("lux", "suggester-name", autocompleteOptions);
|
||||
|
||||
// Suggestions
|
||||
var suggestOptions = new SuggestOptions { UseFuzzyMatching = true };
|
||||
var suggestions = await searchClient.SuggestAsync<Hotel>("lux", "suggester-name", suggestOptions);
|
||||
```
|
||||
|
||||
## Vector Search
|
||||
|
||||
See [references/vector-search.md](references/vector-search.md) for detailed patterns.
|
||||
|
||||
```csharp
|
||||
using Azure.Search.Documents.Models;
|
||||
|
||||
// Pure vector search
|
||||
var vectorQuery = new VectorizedQuery(embedding)
|
||||
{
|
||||
KNearestNeighborsCount = 5,
|
||||
Fields = { "descriptionVector" }
|
||||
};
|
||||
|
||||
var options = new SearchOptions
|
||||
{
|
||||
VectorSearch = new VectorSearchOptions
|
||||
{
|
||||
Queries = { vectorQuery }
|
||||
}
|
||||
};
|
||||
|
||||
var results = await searchClient.SearchAsync<Hotel>(null, options);
|
||||
```
|
||||
|
||||
## Semantic Search
|
||||
|
||||
See [references/semantic-search.md](references/semantic-search.md) for detailed patterns.
|
||||
|
||||
```csharp
|
||||
var options = new SearchOptions
|
||||
{
|
||||
QueryType = SearchQueryType.Semantic,
|
||||
SemanticSearch = new SemanticSearchOptions
|
||||
{
|
||||
SemanticConfigurationName = "my-semantic-config",
|
||||
QueryCaption = new QueryCaption(QueryCaptionType.Extractive),
|
||||
QueryAnswer = new QueryAnswer(QueryAnswerType.Extractive)
|
||||
}
|
||||
};
|
||||
|
||||
var results = await searchClient.SearchAsync<Hotel>("best hotel for families", options);
|
||||
|
||||
// Access semantic answers
|
||||
foreach (var answer in results.Value.SemanticSearch.Answers)
|
||||
{
|
||||
Console.WriteLine($"Answer: {answer.Text} (Score: {answer.Score})");
|
||||
}
|
||||
|
||||
// Access captions
|
||||
await foreach (var result in results.Value.GetResultsAsync())
|
||||
{
|
||||
var caption = result.SemanticSearch?.Captions?.FirstOrDefault();
|
||||
Console.WriteLine($"Caption: {caption?.Text}");
|
||||
}
|
||||
```
|
||||
|
||||
## Hybrid Search (Vector + Keyword + Semantic)
|
||||
|
||||
```csharp
|
||||
var vectorQuery = new VectorizedQuery(embedding)
|
||||
{
|
||||
KNearestNeighborsCount = 5,
|
||||
Fields = { "descriptionVector" }
|
||||
};
|
||||
|
||||
var options = new SearchOptions
|
||||
{
|
||||
QueryType = SearchQueryType.Semantic,
|
||||
SemanticSearch = new SemanticSearchOptions
|
||||
{
|
||||
SemanticConfigurationName = "my-semantic-config"
|
||||
},
|
||||
VectorSearch = new VectorSearchOptions
|
||||
{
|
||||
Queries = { vectorQuery }
|
||||
}
|
||||
};
|
||||
|
||||
// Combines keyword search, vector search, and semantic ranking
|
||||
var results = await searchClient.SearchAsync<Hotel>("luxury beachfront", options);
|
||||
```
|
||||
|
||||
## Field Attributes Reference
|
||||
|
||||
| Attribute | Purpose |
|
||||
|-----------|---------|
|
||||
| `SimpleField` | Non-searchable field (filters, sorting, facets) |
|
||||
| `SearchableField` | Full-text searchable field |
|
||||
| `VectorSearchField` | Vector embedding field |
|
||||
| `IsKey = true` | Document key (required, one per index) |
|
||||
| `IsFilterable = true` | Enable $filter expressions |
|
||||
| `IsSortable = true` | Enable $orderby |
|
||||
| `IsFacetable = true` | Enable faceted navigation |
|
||||
| `IsHidden = true` | Exclude from results |
|
||||
| `AnalyzerName` | Specify text analyzer |
|
||||
|
||||
## Error Handling
|
||||
|
||||
```csharp
|
||||
using Azure;
|
||||
|
||||
try
|
||||
{
|
||||
var results = await searchClient.SearchAsync<Hotel>("query");
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 404)
|
||||
{
|
||||
Console.WriteLine("Index not found");
|
||||
}
|
||||
catch (RequestFailedException ex)
|
||||
{
|
||||
Console.WriteLine($"Search error: {ex.Status} - {ex.ErrorCode}: {ex.Message}");
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use `DefaultAzureCredential`** over API keys for production
|
||||
2. **Use `FieldBuilder`** with model attributes for type-safe index definitions
|
||||
3. **Use `CreateOrUpdateIndexAsync`** for idempotent index creation
|
||||
4. **Batch document operations** for better throughput
|
||||
5. **Use `Select`** to return only needed fields
|
||||
6. **Configure semantic search** for natural language queries
|
||||
7. **Combine vector + keyword + semantic** for best relevance
|
||||
|
||||
## Reference Files
|
||||
|
||||
| File | Contents |
|
||||
|------|----------|
|
||||
| [references/vector-search.md](references/vector-search.md) | Vector search, hybrid search, vectorizers |
|
||||
| [references/semantic-search.md](references/semantic-search.md) | Semantic ranking, captions, answers |
|
||||
265
skills/official/microsoft/dotnet/foundry/voicelive/SKILL.md
Normal file
265
skills/official/microsoft/dotnet/foundry/voicelive/SKILL.md
Normal file
@@ -0,0 +1,265 @@
|
||||
---
|
||||
name: azure-ai-voicelive-dotnet
|
||||
description: |
|
||||
Azure AI Voice Live SDK for .NET. Build real-time voice AI applications with bidirectional WebSocket communication. Use for voice assistants, conversational AI, real-time speech-to-speech, and voice-enabled chatbots. Triggers: "voice live", "real-time voice", "VoiceLiveClient", "VoiceLiveSession", "voice assistant .NET", "bidirectional audio", "speech-to-speech".
|
||||
package: Azure.AI.VoiceLive
|
||||
---
|
||||
|
||||
# Azure.AI.VoiceLive (.NET)
|
||||
|
||||
Real-time voice AI SDK for building bidirectional voice assistants with Azure AI.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
dotnet add package Azure.AI.VoiceLive
|
||||
dotnet add package Azure.Identity
|
||||
dotnet add package NAudio # For audio capture/playback
|
||||
```
|
||||
|
||||
**Current Versions**: Stable v1.0.0, Preview v1.1.0-beta.1
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
AZURE_VOICELIVE_ENDPOINT=https://<resource>.services.ai.azure.com/
|
||||
AZURE_VOICELIVE_MODEL=gpt-4o-realtime-preview
|
||||
AZURE_VOICELIVE_VOICE=en-US-AvaNeural
|
||||
# Optional: API key if not using Entra ID
|
||||
AZURE_VOICELIVE_API_KEY=<your-api-key>
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
### Microsoft Entra ID (Recommended)
|
||||
|
||||
```csharp
|
||||
using Azure.Identity;
|
||||
using Azure.AI.VoiceLive;
|
||||
|
||||
Uri endpoint = new Uri("https://your-resource.cognitiveservices.azure.com");
|
||||
DefaultAzureCredential credential = new DefaultAzureCredential();
|
||||
VoiceLiveClient client = new VoiceLiveClient(endpoint, credential);
|
||||
```
|
||||
|
||||
**Required Role**: `Cognitive Services User` (assign in Azure Portal → Access control)
|
||||
|
||||
### API Key
|
||||
|
||||
```csharp
|
||||
Uri endpoint = new Uri("https://your-resource.cognitiveservices.azure.com");
|
||||
AzureKeyCredential credential = new AzureKeyCredential("your-api-key");
|
||||
VoiceLiveClient client = new VoiceLiveClient(endpoint, credential);
|
||||
```
|
||||
|
||||
## Client Hierarchy
|
||||
|
||||
```
|
||||
VoiceLiveClient
|
||||
└── VoiceLiveSession (WebSocket connection)
|
||||
├── ConfigureSessionAsync()
|
||||
├── GetUpdatesAsync() → SessionUpdate events
|
||||
├── AddItemAsync() → UserMessageItem, FunctionCallOutputItem
|
||||
├── SendAudioAsync()
|
||||
└── StartResponseAsync()
|
||||
```
|
||||
|
||||
## Core Workflow
|
||||
|
||||
### 1. Start Session and Configure
|
||||
|
||||
```csharp
|
||||
using Azure.Identity;
|
||||
using Azure.AI.VoiceLive;
|
||||
|
||||
var endpoint = new Uri(Environment.GetEnvironmentVariable("AZURE_VOICELIVE_ENDPOINT"));
|
||||
var client = new VoiceLiveClient(endpoint, new DefaultAzureCredential());
|
||||
|
||||
var model = "gpt-4o-mini-realtime-preview";
|
||||
|
||||
// Start session
|
||||
using VoiceLiveSession session = await client.StartSessionAsync(model);
|
||||
|
||||
// Configure session
|
||||
VoiceLiveSessionOptions sessionOptions = new()
|
||||
{
|
||||
Model = model,
|
||||
Instructions = "You are a helpful AI assistant. Respond naturally.",
|
||||
Voice = new AzureStandardVoice("en-US-AvaNeural"),
|
||||
TurnDetection = new AzureSemanticVadTurnDetection()
|
||||
{
|
||||
Threshold = 0.5f,
|
||||
PrefixPadding = TimeSpan.FromMilliseconds(300),
|
||||
SilenceDuration = TimeSpan.FromMilliseconds(500)
|
||||
},
|
||||
InputAudioFormat = InputAudioFormat.Pcm16,
|
||||
OutputAudioFormat = OutputAudioFormat.Pcm16
|
||||
};
|
||||
|
||||
// Set modalities (both text and audio for voice assistants)
|
||||
sessionOptions.Modalities.Clear();
|
||||
sessionOptions.Modalities.Add(InteractionModality.Text);
|
||||
sessionOptions.Modalities.Add(InteractionModality.Audio);
|
||||
|
||||
await session.ConfigureSessionAsync(sessionOptions);
|
||||
```
|
||||
|
||||
### 2. Process Events
|
||||
|
||||
```csharp
|
||||
await foreach (SessionUpdate serverEvent in session.GetUpdatesAsync())
|
||||
{
|
||||
switch (serverEvent)
|
||||
{
|
||||
case SessionUpdateResponseAudioDelta audioDelta:
|
||||
byte[] audioData = audioDelta.Delta.ToArray();
|
||||
// Play audio via NAudio or other audio library
|
||||
break;
|
||||
|
||||
case SessionUpdateResponseTextDelta textDelta:
|
||||
Console.Write(textDelta.Delta);
|
||||
break;
|
||||
|
||||
case SessionUpdateResponseFunctionCallArgumentsDone functionCall:
|
||||
// Handle function call (see Function Calling section)
|
||||
break;
|
||||
|
||||
case SessionUpdateError error:
|
||||
Console.WriteLine($"Error: {error.Error.Message}");
|
||||
break;
|
||||
|
||||
case SessionUpdateResponseDone:
|
||||
Console.WriteLine("\n--- Response complete ---");
|
||||
break;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Send User Message
|
||||
|
||||
```csharp
|
||||
await session.AddItemAsync(new UserMessageItem("Hello, can you help me?"));
|
||||
await session.StartResponseAsync();
|
||||
```
|
||||
|
||||
### 4. Function Calling
|
||||
|
||||
```csharp
|
||||
// Define function
|
||||
var weatherFunction = new VoiceLiveFunctionDefinition("get_current_weather")
|
||||
{
|
||||
Description = "Get the current weather for a given location",
|
||||
Parameters = BinaryData.FromString("""
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state or country"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
""")
|
||||
};
|
||||
|
||||
// Add to session options
|
||||
sessionOptions.Tools.Add(weatherFunction);
|
||||
|
||||
// Handle function call in event loop
|
||||
if (serverEvent is SessionUpdateResponseFunctionCallArgumentsDone functionCall)
|
||||
{
|
||||
if (functionCall.Name == "get_current_weather")
|
||||
{
|
||||
var parameters = JsonSerializer.Deserialize<Dictionary<string, string>>(functionCall.Arguments);
|
||||
string location = parameters?["location"] ?? "";
|
||||
|
||||
// Call external service
|
||||
string weatherInfo = $"The weather in {location} is sunny, 75°F.";
|
||||
|
||||
// Send response
|
||||
await session.AddItemAsync(new FunctionCallOutputItem(functionCall.CallId, weatherInfo));
|
||||
await session.StartResponseAsync();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Voice Options
|
||||
|
||||
| Voice Type | Class | Example |
|
||||
|------------|-------|---------|
|
||||
| Azure Standard | `AzureStandardVoice` | `"en-US-AvaNeural"` |
|
||||
| Azure HD | `AzureStandardVoice` | `"en-US-Ava:DragonHDLatestNeural"` |
|
||||
| Azure Custom | `AzureCustomVoice` | Custom voice with endpoint ID |
|
||||
|
||||
## Supported Models
|
||||
|
||||
| Model | Description |
|
||||
|-------|-------------|
|
||||
| `gpt-4o-realtime-preview` | GPT-4o with real-time audio |
|
||||
| `gpt-4o-mini-realtime-preview` | Lightweight, fast interactions |
|
||||
| `phi4-mm-realtime` | Cost-effective multimodal |
|
||||
|
||||
## Key Types Reference
|
||||
|
||||
| Type | Purpose |
|
||||
|------|---------|
|
||||
| `VoiceLiveClient` | Main client for creating sessions |
|
||||
| `VoiceLiveSession` | Active WebSocket session |
|
||||
| `VoiceLiveSessionOptions` | Session configuration |
|
||||
| `AzureStandardVoice` | Standard Azure voice provider |
|
||||
| `AzureSemanticVadTurnDetection` | Voice activity detection |
|
||||
| `VoiceLiveFunctionDefinition` | Function tool definition |
|
||||
| `UserMessageItem` | User text message |
|
||||
| `FunctionCallOutputItem` | Function call response |
|
||||
| `SessionUpdateResponseAudioDelta` | Audio chunk event |
|
||||
| `SessionUpdateResponseTextDelta` | Text chunk event |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always set both modalities** — Include `Text` and `Audio` for voice assistants
|
||||
2. **Use `AzureSemanticVadTurnDetection`** — Provides natural conversation flow
|
||||
3. **Configure appropriate silence duration** — 500ms typical to avoid premature cutoffs
|
||||
4. **Use `using` statement** — Ensures proper session disposal
|
||||
5. **Handle all event types** — Check for errors, audio, text, and function calls
|
||||
6. **Use DefaultAzureCredential** — Never hardcode API keys
|
||||
|
||||
## Error Handling
|
||||
|
||||
```csharp
|
||||
if (serverEvent is SessionUpdateError error)
|
||||
{
|
||||
if (error.Error.Message.Contains("Cancellation failed: no active response"))
|
||||
{
|
||||
// Benign error, can ignore
|
||||
}
|
||||
else
|
||||
{
|
||||
Console.WriteLine($"Error: {error.Error.Message}");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Audio Configuration
|
||||
|
||||
- **Input Format**: `InputAudioFormat.Pcm16` (16-bit PCM)
|
||||
- **Output Format**: `OutputAudioFormat.Pcm16`
|
||||
- **Sample Rate**: 24kHz recommended
|
||||
- **Channels**: Mono
|
||||
|
||||
## Related SDKs
|
||||
|
||||
| SDK | Purpose | Install |
|
||||
|-----|---------|---------|
|
||||
| `Azure.AI.VoiceLive` | Real-time voice (this SDK) | `dotnet add package Azure.AI.VoiceLive` |
|
||||
| `Microsoft.CognitiveServices.Speech` | Speech-to-text, text-to-speech | `dotnet add package Microsoft.CognitiveServices.Speech` |
|
||||
| `NAudio` | Audio capture/playback | `dotnet add package NAudio` |
|
||||
|
||||
## Reference Links
|
||||
|
||||
| Resource | URL |
|
||||
|----------|-----|
|
||||
| NuGet Package | https://www.nuget.org/packages/Azure.AI.VoiceLive |
|
||||
| API Reference | https://learn.microsoft.com/dotnet/api/azure.ai.voicelive |
|
||||
| GitHub Source | https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/ai/Azure.AI.VoiceLive |
|
||||
| Quickstart | https://learn.microsoft.com/azure/ai-services/speech-service/voice-live-quickstart |
|
||||
@@ -0,0 +1,328 @@
|
||||
---
|
||||
name: azure-mgmt-weightsandbiases-dotnet
|
||||
description: |
|
||||
Azure Weights & Biases SDK for .NET. ML experiment tracking and model management via Azure Marketplace. Use for creating W&B instances, managing SSO, marketplace integration, and ML observability. Triggers: "Weights and Biases", "W&B", "WeightsAndBiases", "ML experiment tracking", "model registry", "experiment management", "wandb".
|
||||
package: Azure.ResourceManager.WeightsAndBiases
|
||||
---
|
||||
|
||||
# Azure.ResourceManager.WeightsAndBiases (.NET)
|
||||
|
||||
Azure Resource Manager SDK for deploying and managing Weights & Biases ML experiment tracking instances via Azure Marketplace.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
dotnet add package Azure.ResourceManager.WeightsAndBiases --prerelease
|
||||
dotnet add package Azure.Identity
|
||||
```
|
||||
|
||||
**Current Version**: v1.0.0-beta.1 (preview)
|
||||
**API Version**: 2024-09-18-preview
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
AZURE_SUBSCRIPTION_ID=<your-subscription-id>
|
||||
AZURE_RESOURCE_GROUP=<your-resource-group>
|
||||
AZURE_WANDB_INSTANCE_NAME=<your-wandb-instance>
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
```csharp
|
||||
using Azure.Identity;
|
||||
using Azure.ResourceManager;
|
||||
using Azure.ResourceManager.WeightsAndBiases;
|
||||
|
||||
ArmClient client = new ArmClient(new DefaultAzureCredential());
|
||||
```
|
||||
|
||||
## Resource Hierarchy
|
||||
|
||||
```
|
||||
Subscription
|
||||
└── ResourceGroup
|
||||
└── WeightsAndBiasesInstance # W&B deployment from Azure Marketplace
|
||||
├── Properties
|
||||
│ ├── Marketplace # Offer details, plan, publisher
|
||||
│ ├── User # Admin user info
|
||||
│ ├── PartnerProperties # W&B-specific config (region, subdomain)
|
||||
│ └── SingleSignOnPropertiesV2 # Entra ID SSO configuration
|
||||
└── Identity # Managed identity (optional)
|
||||
```
|
||||
|
||||
## Core Workflows
|
||||
|
||||
### 1. Create Weights & Biases Instance
|
||||
|
||||
```csharp
|
||||
using Azure.ResourceManager.WeightsAndBiases;
|
||||
using Azure.ResourceManager.WeightsAndBiases.Models;
|
||||
|
||||
ResourceGroupResource resourceGroup = await client
|
||||
.GetDefaultSubscriptionAsync()
|
||||
.Result
|
||||
.GetResourceGroupAsync("my-resource-group");
|
||||
|
||||
WeightsAndBiasesInstanceCollection instances = resourceGroup.GetWeightsAndBiasesInstances();
|
||||
|
||||
WeightsAndBiasesInstanceData data = new WeightsAndBiasesInstanceData(AzureLocation.EastUS)
|
||||
{
|
||||
Properties = new WeightsAndBiasesInstanceProperties
|
||||
{
|
||||
// Marketplace configuration
|
||||
Marketplace = new WeightsAndBiasesMarketplaceDetails
|
||||
{
|
||||
SubscriptionId = "<marketplace-subscription-id>",
|
||||
OfferDetails = new WeightsAndBiasesOfferDetails
|
||||
{
|
||||
PublisherId = "wandb",
|
||||
OfferId = "wandb-pay-as-you-go",
|
||||
PlanId = "wandb-payg",
|
||||
PlanName = "Pay As You Go",
|
||||
TermId = "monthly",
|
||||
TermUnit = "P1M"
|
||||
}
|
||||
},
|
||||
// Admin user
|
||||
User = new WeightsAndBiasesUserDetails
|
||||
{
|
||||
FirstName = "Admin",
|
||||
LastName = "User",
|
||||
EmailAddress = "admin@example.com",
|
||||
Upn = "admin@example.com"
|
||||
},
|
||||
// W&B-specific configuration
|
||||
PartnerProperties = new WeightsAndBiasesPartnerProperties
|
||||
{
|
||||
Region = WeightsAndBiasesRegion.EastUS,
|
||||
Subdomain = "my-company-wandb"
|
||||
}
|
||||
},
|
||||
// Optional: Enable managed identity
|
||||
Identity = new ManagedServiceIdentity(ManagedServiceIdentityType.SystemAssigned)
|
||||
};
|
||||
|
||||
ArmOperation<WeightsAndBiasesInstanceResource> operation = await instances
|
||||
.CreateOrUpdateAsync(WaitUntil.Completed, "my-wandb-instance", data);
|
||||
|
||||
WeightsAndBiasesInstanceResource instance = operation.Value;
|
||||
|
||||
Console.WriteLine($"W&B Instance created: {instance.Data.Name}");
|
||||
Console.WriteLine($"Provisioning state: {instance.Data.Properties.ProvisioningState}");
|
||||
```
|
||||
|
||||
### 2. Get Existing Instance
|
||||
|
||||
```csharp
|
||||
WeightsAndBiasesInstanceResource instance = await resourceGroup
|
||||
.GetWeightsAndBiasesInstanceAsync("my-wandb-instance");
|
||||
|
||||
Console.WriteLine($"Instance: {instance.Data.Name}");
|
||||
Console.WriteLine($"Location: {instance.Data.Location}");
|
||||
Console.WriteLine($"State: {instance.Data.Properties.ProvisioningState}");
|
||||
|
||||
if (instance.Data.Properties.PartnerProperties != null)
|
||||
{
|
||||
Console.WriteLine($"Region: {instance.Data.Properties.PartnerProperties.Region}");
|
||||
Console.WriteLine($"Subdomain: {instance.Data.Properties.PartnerProperties.Subdomain}");
|
||||
}
|
||||
```
|
||||
|
||||
### 3. List All Instances
|
||||
|
||||
```csharp
|
||||
// List in resource group
|
||||
await foreach (WeightsAndBiasesInstanceResource instance in
|
||||
resourceGroup.GetWeightsAndBiasesInstances())
|
||||
{
|
||||
Console.WriteLine($"Instance: {instance.Data.Name}");
|
||||
Console.WriteLine($" Location: {instance.Data.Location}");
|
||||
Console.WriteLine($" State: {instance.Data.Properties.ProvisioningState}");
|
||||
}
|
||||
|
||||
// List in subscription
|
||||
SubscriptionResource subscription = await client.GetDefaultSubscriptionAsync();
|
||||
await foreach (WeightsAndBiasesInstanceResource instance in
|
||||
subscription.GetWeightsAndBiasesInstancesAsync())
|
||||
{
|
||||
Console.WriteLine($"{instance.Data.Name} in {instance.Id.ResourceGroupName}");
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Configure Single Sign-On (SSO)
|
||||
|
||||
```csharp
|
||||
WeightsAndBiasesInstanceResource instance = await resourceGroup
|
||||
.GetWeightsAndBiasesInstanceAsync("my-wandb-instance");
|
||||
|
||||
// Update with SSO configuration
|
||||
WeightsAndBiasesInstanceData updateData = instance.Data;
|
||||
|
||||
updateData.Properties.SingleSignOnPropertiesV2 = new WeightsAndBiasSingleSignOnPropertiesV2
|
||||
{
|
||||
Type = WeightsAndBiasSingleSignOnType.Saml,
|
||||
State = WeightsAndBiasSingleSignOnState.Enable,
|
||||
EnterpriseAppId = "<entra-app-id>",
|
||||
AadDomains = { "example.com", "contoso.com" }
|
||||
};
|
||||
|
||||
ArmOperation<WeightsAndBiasesInstanceResource> operation = await resourceGroup
|
||||
.GetWeightsAndBiasesInstances()
|
||||
.CreateOrUpdateAsync(WaitUntil.Completed, "my-wandb-instance", updateData);
|
||||
```
|
||||
|
||||
### 5. Update Instance
|
||||
|
||||
```csharp
|
||||
WeightsAndBiasesInstanceResource instance = await resourceGroup
|
||||
.GetWeightsAndBiasesInstanceAsync("my-wandb-instance");
|
||||
|
||||
// Update tags
|
||||
WeightsAndBiasesInstancePatch patch = new WeightsAndBiasesInstancePatch
|
||||
{
|
||||
Tags =
|
||||
{
|
||||
{ "environment", "production" },
|
||||
{ "team", "ml-platform" },
|
||||
{ "costCenter", "CC-ML-001" }
|
||||
}
|
||||
};
|
||||
|
||||
instance = await instance.UpdateAsync(patch);
|
||||
Console.WriteLine($"Updated instance: {instance.Data.Name}");
|
||||
```
|
||||
|
||||
### 6. Delete Instance
|
||||
|
||||
```csharp
|
||||
WeightsAndBiasesInstanceResource instance = await resourceGroup
|
||||
.GetWeightsAndBiasesInstanceAsync("my-wandb-instance");
|
||||
|
||||
await instance.DeleteAsync(WaitUntil.Completed);
|
||||
Console.WriteLine("Instance deleted");
|
||||
```
|
||||
|
||||
### 7. Check Resource Name Availability
|
||||
|
||||
```csharp
|
||||
// Check if name is available before creating
|
||||
// (Implement via direct ARM call if SDK doesn't expose this)
|
||||
try
|
||||
{
|
||||
await resourceGroup.GetWeightsAndBiasesInstanceAsync("desired-name");
|
||||
Console.WriteLine("Name is already taken");
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 404)
|
||||
{
|
||||
Console.WriteLine("Name is available");
|
||||
}
|
||||
```
|
||||
|
||||
## Key Types Reference
|
||||
|
||||
| Type | Purpose |
|
||||
|------|---------|
|
||||
| `WeightsAndBiasesInstanceResource` | W&B instance resource |
|
||||
| `WeightsAndBiasesInstanceData` | Instance configuration data |
|
||||
| `WeightsAndBiasesInstanceCollection` | Collection of instances |
|
||||
| `WeightsAndBiasesInstanceProperties` | Instance properties |
|
||||
| `WeightsAndBiasesMarketplaceDetails` | Marketplace subscription info |
|
||||
| `WeightsAndBiasesOfferDetails` | Marketplace offer details |
|
||||
| `WeightsAndBiasesUserDetails` | Admin user information |
|
||||
| `WeightsAndBiasesPartnerProperties` | W&B-specific configuration |
|
||||
| `WeightsAndBiasSingleSignOnPropertiesV2` | SSO configuration |
|
||||
| `WeightsAndBiasesInstancePatch` | Patch for updates |
|
||||
| `WeightsAndBiasesRegion` | Supported regions enum |
|
||||
|
||||
## Available Regions
|
||||
|
||||
| Region Enum | Azure Region |
|
||||
|-------------|--------------|
|
||||
| `WeightsAndBiasesRegion.EastUS` | East US |
|
||||
| `WeightsAndBiasesRegion.CentralUS` | Central US |
|
||||
| `WeightsAndBiasesRegion.WestUS` | West US |
|
||||
| `WeightsAndBiasesRegion.WestEurope` | West Europe |
|
||||
| `WeightsAndBiasesRegion.JapanEast` | Japan East |
|
||||
| `WeightsAndBiasesRegion.KoreaCentral` | Korea Central |
|
||||
|
||||
## Marketplace Offer Details
|
||||
|
||||
For Azure Marketplace integration:
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Publisher ID | `wandb` |
|
||||
| Offer ID | `wandb-pay-as-you-go` |
|
||||
| Plan ID | `wandb-payg` (Pay As You Go) |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use DefaultAzureCredential** — Supports multiple auth methods automatically
|
||||
2. **Enable managed identity** — For secure access to other Azure resources
|
||||
3. **Configure SSO** — Enable Entra ID SSO for enterprise security
|
||||
4. **Tag resources** — Use tags for cost tracking and organization
|
||||
5. **Check provisioning state** — Wait for `Succeeded` before using instance
|
||||
6. **Use appropriate region** — Choose region closest to your compute
|
||||
7. **Monitor with Azure** — Use Azure Monitor for resource health
|
||||
|
||||
## Error Handling
|
||||
|
||||
```csharp
|
||||
using Azure;
|
||||
|
||||
try
|
||||
{
|
||||
ArmOperation<WeightsAndBiasesInstanceResource> operation = await instances
|
||||
.CreateOrUpdateAsync(WaitUntil.Completed, "my-wandb", data);
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 409)
|
||||
{
|
||||
Console.WriteLine("Instance already exists or name conflict");
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 400)
|
||||
{
|
||||
Console.WriteLine($"Invalid configuration: {ex.Message}");
|
||||
}
|
||||
catch (RequestFailedException ex)
|
||||
{
|
||||
Console.WriteLine($"Azure error: {ex.Status} - {ex.Message}");
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with W&B SDK
|
||||
|
||||
After creating the Azure resource, use the W&B Python SDK for experiment tracking:
|
||||
|
||||
```python
|
||||
# Install: pip install wandb
|
||||
import wandb
|
||||
|
||||
# Login with your W&B API key from the Azure-deployed instance
|
||||
wandb.login(host="https://my-company-wandb.wandb.ai")
|
||||
|
||||
# Initialize a run
|
||||
run = wandb.init(project="my-ml-project")
|
||||
|
||||
# Log metrics
|
||||
wandb.log({"accuracy": 0.95, "loss": 0.05})
|
||||
|
||||
# Finish run
|
||||
run.finish()
|
||||
```
|
||||
|
||||
## Related SDKs
|
||||
|
||||
| SDK | Purpose | Install |
|
||||
|-----|---------|---------|
|
||||
| `Azure.ResourceManager.WeightsAndBiases` | W&B instance management (this SDK) | `dotnet add package Azure.ResourceManager.WeightsAndBiases --prerelease` |
|
||||
| `Azure.ResourceManager.MachineLearning` | Azure ML workspaces | `dotnet add package Azure.ResourceManager.MachineLearning` |
|
||||
|
||||
## Reference Links
|
||||
|
||||
| Resource | URL |
|
||||
|----------|-----|
|
||||
| NuGet Package | https://www.nuget.org/packages/Azure.ResourceManager.WeightsAndBiases |
|
||||
| W&B Documentation | https://docs.wandb.ai/ |
|
||||
| Azure Marketplace | https://azuremarketplace.microsoft.com/marketplace/apps/wandb.wandb-pay-as-you-go |
|
||||
| GitHub Source | https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/weightsandbiases |
|
||||
Reference in New Issue
Block a user