feat: Update scripts and documentation to support new parameters for Azure services and Content Understanding API

Author: Pavan-Microsoft

documents/CustomizeData.md

@@ -26,11 +26,11 @@ If you would like to update the solution to leverage your own data please follow
     bash ./infra/scripts/process_custom_data.sh \
       <Resource-Group-Name> <Azure-Subscription-ID> \
       <Storage-Account-Name> <Storage-Container-Name> \
-      <SQL-Server-Name> <SQL-Database-Name> <SQL-Managed-Identity-Client-ID> <SQL-Managed-Identity-Display-Name> \
+      <SQL-Server-Name> <SQL-Database-Name> <Backend-User-MID-Client-ID> <Backend-User-MID-Display-Name> \
       <AI-Search-Name> <Search-Endpoint> \
       <AI-Foundry-Resource-ID> <CU-Foundry-Resource-ID> \
-      <OpenAI-Endpoint> <Embedding-Model> <Deployment-Model> <OpenAI-Preview-API-Version> \
-      <CU-Endpoint> <AI-Agent-Endpoint>
+      <OpenAI-Endpoint> <Embedding-Model> <Deployment-Model> \
+      <CU-Endpoint> <AI-Agent-Endpoint> <CU-API-Version>
     ```
 
 ## How to Login to VM Using Azure Bastion

documents/DeploymentGuide.md

@@ -138,7 +138,7 @@ If you're not using one of the above options for opening the project, then you'l
 1. Make sure the following tools are installed:
     - [PowerShell](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.5) <small>(v7.0+)</small> - available for Windows, macOS, and Linux.
     - [Azure Developer CLI (azd)](https://aka.ms/install-azd) <small>(v1.18.0+)</small> - version
-    - [Python 3.9+](https://www.python.org/downloads/)
+    - [Python 3.9 to 3.11](https://www.python.org/downloads/)
     - [Docker Desktop](https://www.docker.com/products/docker-desktop/)
     - [Git](https://git-scm.com/downloads)
 
@@ -250,11 +250,11 @@ Once you've opened the project in [Codespaces](#github-codespaces), [Dev Contain
     bash ./infra/scripts/process_sample_data.sh \
       <Resource-Group-Name> <Azure-Subscription-ID> \
       <Storage-Account-Name> <Storage-Container-Name> \
-      <SQL-Server-Name> <SQL-Database-Name> <SQL-Managed-Identity-Client-ID> <SQL-Managed-Identity-Display-Name> \
+      <SQL-Server-Name> <SQL-Database-Name> <Backend-User-MID-Client-ID> <Backend-User-MID-Display-Name> \
       <AI-Search-Name> <Search-Endpoint> \
       <AI-Foundry-Resource-ID> <CU-Foundry-Resource-ID> \
-      <OpenAI-Endpoint> <Embedding-Model> <Deployment-Model> <OpenAI-Preview-API-Version> \
-      <CU-Endpoint> <AI-Agent-Endpoint>
+      <OpenAI-Endpoint> <Embedding-Model> <Deployment-Model> \
+      <CU-Endpoint> <AI-Agent-Endpoint> <CU-API-Version>
     ```
 
 6. Once the deployment has completed successfully, open the [Azure Portal](https://portal.azure.com/), go to the deployed resource group, find the App Service, and get the app URL from `Default domain`.

documents/LocalDebuggingSetup.md

@@ -11,7 +11,7 @@ Install these tools before you start:
   - [Azure Tools](https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-node-azure-pack)
   - [Bicep](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-bicep)
   - [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
-- [Python 3.11](https://www.python.org/downloads/). **Important:** Check "Add Python to PATH" during installation.
+- [Python 3.9 to 3.11](https://www.python.org/downloads/). **Important:** Check "Add Python to PATH" during installation.
 - [PowerShell 7.0+](https://github.com/PowerShell/PowerShell#get-powershell).
 - [Node.js (LTS)](https://nodejs.org/en).
 - [Git](https://git-scm.com/downloads).

infra/main.bicep

@@ -77,6 +77,9 @@ param azureOpenAIApiVersion string = '2025-01-01-preview'
 @description('Optional. Version of AI Agent API.')
 param azureAiAgentApiVersion string = '2025-05-01'
 
+@description('Optional. Version of Content Understanding API.')
+param azureContentUnderstandingApiVersion string = '2024-12-01-preview'
+
 // You can increase this, but capacity is limited per model/region, so you will get errors if you go over
 // https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits
 @minValue(10)
@@ -1428,6 +1431,9 @@ output AZURE_OPENAI_API_VERSION string = azureOpenAIApiVersion
 @description('Contains Azure OpenAI Preview API version.')
 output AZURE_OPENAI_PREVIEW_API_VERSION string = azureAiAgentApiVersion
 
+@description('Contains Content Understanding API version.')
+output AZURE_CONTENT_UNDERSTANDING_API_VERSION string = azureContentUnderstandingApiVersion
+
 @description('Contains Azure OpenAI resource name.')
 output AZURE_OPENAI_RESOURCE string = aiFoundryAiServices.outputs.name
 
@@ -1440,11 +1446,11 @@ output SQLDB_DATABASE string = 'sqldb-${solutionSuffix}'
 @description('Contains SQL server name.')
 output SQLDB_SERVER string = '${sqlDBModule.outputs.name }${environment().suffixes.sqlServerHostname}'
 
-@description('Display name of the SQL managed identity.')
-output SQLDB_USER_MID_NAME string = backendUserAssignedIdentity.outputs.name
+@description('Display name of the backend API user-assigned managed identity (also used for SQL database access).')
+output BACKEND_USER_MID_NAME string = backendUserAssignedIdentity.outputs.name
 
-@description('Contains SQL database user managed identity client ID.')
-output SQLDB_USER_MID string = backendUserAssignedIdentity.outputs.clientId
+@description('Client ID of the backend API user-assigned managed identity (also used for SQL database access).')
+output BACKEND_USER_MID string = backendUserAssignedIdentity.outputs.clientId
 
 @description('Contains AI project client usage setting.')
 output USE_AI_PROJECT_CLIENT string = 'False'

infra/scripts/copy_kb_files.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
 # Variables
-storageAccount="$1"
+storageAccountName="$1"
 containerName="$2"
 resourceGroupName="$3"
 
@@ -12,14 +12,14 @@ zipFileName2="infra/data/audio_data.zip"
 extractedFolder2="audio_data"
 
 echo "Script Started"
-echo "Storage Account: $storageAccount"
+echo "Storage Account: $storageAccountName"
 echo "Container Name: $containerName"
 echo "Resource Group: $resourceGroupName"
 
 # Validate required parameters
-if [ -z "$storageAccount" ] || [ -z "$containerName" ] || [ -z "$resourceGroupName" ]; then
+if [ -z "$storageAccountName" ] || [ -z "$containerName" ] || [ -z "$resourceGroupName" ]; then
     echo "Error: Missing required parameters."
-    echo "Usage: $0 <storageAccount> <containerName> <resourceGroupName>"
+    echo "Usage: $0 <storageAccountName> <containerName> <resourceGroupName>"
     exit 1
 fi
 
@@ -49,7 +49,7 @@ fi
 # Check and assign Storage Blob Data Contributor role to current user
 echo "Checking Storage Blob Data Contributor role assignment..."
 signed_user_id=$(az ad signed-in-user show --query id --output tsv)
-storage_resource_id=$(az storage account show --name "$storageAccount" --resource-group "$resourceGroupName" --query id --output tsv)
+storage_resource_id=$(az storage account show --name "$storageAccountName" --resource-group "$resourceGroupName" --query id --output tsv)
 
 role_assignment=$(MSYS_NO_PATHCONV=1 az role assignment list --assignee $signed_user_id --role "Storage Blob Data Contributor" --scope $storage_resource_id --query "[].roleDefinitionId" -o tsv)
 if [ -z "$role_assignment" ]; then
@@ -73,7 +73,7 @@ fi
 echo "Uploading call transcripts to storage account..."
 if [ -d "$extractedFolder1" ]; then
 	az storage blob upload-batch \
-		--account-name "$storageAccount" \
+		--account-name "$storageAccountName" \
 		--destination "$containerName/$extractedFolder1" \
 		--source "$extractedFolder1" \
 		--auth-mode login \
@@ -92,7 +92,7 @@ fi
 echo "Uploading audio data to storage account..."
 if [ -d "$extractedFolder2" ]; then
 	az storage blob upload-batch \
-		--account-name "$storageAccount" \
+		--account-name "$storageAccountName" \
 		--destination "$containerName/$extractedFolder2" \
 		--source "$extractedFolder2" \
 		--auth-mode login \
@@ -111,13 +111,13 @@ fi
 # Create custom data directories for user uploads
 echo "Creating custom data directories..."
 az storage fs directory create \
-	--account-name "$storageAccount" \
+	--account-name "$storageAccountName" \
 	--file-system "$containerName" \
 	--name custom_audiodata \
 	--auth-mode login 2>/dev/null || echo "custom_audiodata directory may already exist"
 
 az storage fs directory create \
-	--account-name "$storageAccount" \
+	--account-name "$storageAccountName" \
 	--file-system "$containerName" \
 	--name custom_transcripts \
 	--auth-mode login 2>/dev/null || echo "custom_transcripts directory may already exist"

infra/scripts/index_scripts/02_create_cu_template_audio.py

@@ -9,24 +9,25 @@
 # Get parameters from command line
 p = argparse.ArgumentParser()
 p.add_argument("--cu_endpoint", required=True)
+p.add_argument("--cu_api_version", required=True)
 args = p.parse_args()
 
 CU_ENDPOINT = args.cu_endpoint
+CU_API_VERSION = args.cu_api_version
 
-AZURE_AI_API_VERSION = "2024-12-01-preview"
 ANALYZER_ID = "ckm-audio"
 
 ANALYZER_TEMPLATE_FILE = 'infra/data/ckm-analyzer_config_audio.json'
 
-# Add parent directory to path for module access
+# Add parent directory to path for imports
 sys.path.append(str(Path.cwd().parent))
 
 credential = AzureCliCredential()
 # Initialize Content Understanding Client
 token_provider = get_bearer_token_provider(credential, "https://cognitiveservices.azure.com/.default")
 client = AzureContentUnderstandingClient(
     endpoint=CU_ENDPOINT,
-    api_version=AZURE_AI_API_VERSION,
+    api_version=CU_API_VERSION,
     token_provider=token_provider
 )
 

infra/scripts/index_scripts/02_create_cu_template_text.py

@@ -7,11 +7,12 @@
 # Get parameters from command line
 p = argparse.ArgumentParser()
 p.add_argument("--cu_endpoint", required=True)
+p.add_argument("--cu_api_version", required=True)
 args = p.parse_args()
 
 CU_ENDPOINT = args.cu_endpoint
+CU_API_VERSION = args.cu_api_version
 
-AZURE_AI_API_VERSION = "2024-12-01-preview"
 ANALYZER_ID = "ckm-json"
 
 ANALYZER_TEMPLATE_FILE = 'infra/data/ckm-analyzer_config_text.json'
@@ -21,7 +22,7 @@
 token_provider = get_bearer_token_provider(credential, "https://cognitiveservices.azure.com/.default")
 client = AzureContentUnderstandingClient(
     endpoint=CU_ENDPOINT,
-    api_version=AZURE_AI_API_VERSION,
+    api_version=CU_API_VERSION,
     token_provider=token_provider
 )
 

infra/scripts/index_scripts/03_cu_process_data_text.py

@@ -22,24 +22,24 @@
 p = argparse.ArgumentParser()
 p.add_argument("--search_endpoint", required=True)
 p.add_argument("--ai_project_endpoint", required=True)
-p.add_argument("--openai_api_version", required=True)
 p.add_argument("--deployment_model", required=True)
 p.add_argument("--embedding_model", required=True)
-p.add_argument("--storage_account", required=True)
+p.add_argument("--storage_account_name", required=True)
 p.add_argument("--sql_server", required=True)
 p.add_argument("--sql_database", required=True)
 p.add_argument("--cu_endpoint", required=True)
+p.add_argument("--cu_api_version", required=True)
 args = p.parse_args()
 
 SEARCH_ENDPOINT = args.search_endpoint
 AI_PROJECT_ENDPOINT = args.ai_project_endpoint
-OPENAI_API_VERSION = args.openai_api_version
 DEPLOYMENT_MODEL = args.deployment_model
 EMBEDDING_MODEL = args.embedding_model
-STORAGE_ACCOUNT = args.storage_account
+STORAGE_ACCOUNT_NAME = args.storage_account_name
 SQL_SERVER = args.sql_server
 SQL_DATABASE = args.sql_database
 CU_ENDPOINT = args.cu_endpoint
+CU_API_VERSION = args.cu_api_version
 
 FILE_SYSTEM_CLIENT_NAME = "data"
 DIRECTORY = 'call_transcripts'
@@ -52,7 +52,7 @@
 print("Parameters received.")
 
 # Azure DataLake setup
-account_url = f"https://{STORAGE_ACCOUNT}.dfs.core.windows.net"
+account_url = f"https://{STORAGE_ACCOUNT_NAME}.dfs.core.windows.net"
 credential = AzureCliCredential()
 service_client = DataLakeServiceClient(account_url, credential=credential, api_version='2023-01-03')
 file_system_client = service_client.get_file_system_client(FILE_SYSTEM_CLIENT_NAME)
@@ -76,10 +76,6 @@
 cursor = conn.cursor()
 print("SQL Server connection established.")
 
-# CU API setup
-AZURE_AI_API_VERSION = "2024-12-01-preview"
-print("Setup complete.")
-
 # SQL data type mapping for pandas to SQL conversion
 sql_data_types = {
     'int64': 'INT',
@@ -170,7 +166,7 @@ def generate_sql_insert_script(df, table_name, columns, sql_file_name):
 cu_token_provider = get_bearer_token_provider(cu_credential, "https://cognitiveservices.azure.com/.default")
 cu_client = AzureContentUnderstandingClient(
     endpoint=CU_ENDPOINT,
-    api_version=AZURE_AI_API_VERSION,
+    api_version=CU_API_VERSION,
     token_provider=cu_token_provider
 )
 ANALYZER_ID = "ckm-json"

infra/scripts/index_scripts/04_cu_process_custom_data.py

@@ -43,27 +43,27 @@
 parser.add_argument('--search_endpoint', required=True, help='Azure AI Search endpoint')
 parser.add_argument('--openai_endpoint', required=True, help='Azure OpenAI endpoint')
 parser.add_argument('--ai_project_endpoint', required=True, help='Azure AI Project endpoint')
-parser.add_argument('--openai_api_version', required=True, help='Azure OpenAI API version')
 parser.add_argument('--deployment_model', required=True, help='Azure OpenAI deployment model name')
 parser.add_argument('--embedding_model', required=True, help='Azure OpenAI embedding model name')
-parser.add_argument('--storage_account', required=True, help='Azure Storage Account name')
+parser.add_argument('--storage_account_name', required=True, help='Azure Storage Account name')
 parser.add_argument('--sql_server', required=True, help='Azure SQL Server name')
 parser.add_argument('--sql_database', required=True, help='Azure SQL Database name')
 parser.add_argument('--cu_endpoint', required=True, help='Azure Content Understanding endpoint')
+parser.add_argument('--cu_api_version', required=True, help='Azure Content Understanding API version')
 
 args = parser.parse_args()
 
 # Assign arguments to variables
 SEARCH_ENDPOINT = args.search_endpoint
 OPENAI_ENDPOINT = args.openai_endpoint
 AI_PROJECT_ENDPOINT = args.ai_project_endpoint
-AZURE_AI_API_VERSION = args.openai_api_version
 DEPLOYMENT_MODEL = args.deployment_model
 EMBEDDING_MODEL = args.embedding_model
-STORAGE_ACCOUNT_NAME = args.storage_account
+STORAGE_ACCOUNT_NAME = args.storage_account_name
 SQL_SERVER = args.sql_server
 SQL_DATABASE = args.sql_database
 CU_ENDPOINT = args.cu_endpoint
+CU_API_VERSION = args.cu_api_version
 
 print("Command-line arguments parsed.")
 
@@ -185,12 +185,11 @@ def create_search_index():
 print("SQL Server connection established.")
 
 # Content Understanding client
-AZURE_AI_API_VERSION = "2024-12-01-preview"
 cu_credential = AzureCliCredential()
 cu_token_provider = get_bearer_token_provider(cu_credential, "https://cognitiveservices.azure.com/.default")
 cu_client = AzureContentUnderstandingClient(
     endpoint=CU_ENDPOINT,
-    api_version=AZURE_AI_API_VERSION,
+    api_version=CU_API_VERSION,
     token_provider=cu_token_provider
 )
 print("Content Understanding client initialized.")