chore: dev to main (Improve Logging & Telemetry, Standardize Deployment Parameters)

[Pavan-Microsoft]

Purpose

This pull request introduces improvements to logging, error tracking, and parameter naming consistency across the API and infrastructure scripts. The main focus is on enhancing observability (with more granular logging and event tracking), standardizing parameter names in deployment files, and making script parsing more robust.

Observability & Logging Enhancements:

• Added detailed logging and event tracking for API endpoints in api_routes.py and history_routes.py, including error events and successful operations, to improve traceability and debugging. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
• Removed redundant Application Insights configuration from api_routes.py and history_routes.py, centralizing telemetry setup. [1] [2]

Parameter Naming Consistency:

• Standardized parameter names in main.parameters.json and main.waf.parameters.json for deployment type, capacity, and VM admin credentials to improve clarity and maintainability. [1] [2] [3] [4] [5]

Script Robustness Improvements:

• Updated Bash scripts (process_custom_data.sh, process_sample_data.sh, run_create_agents_scripts.sh) to use case-insensitive key matching when parsing deployment outputs, reducing potential parsing errors. [1] [2] [3]

Infrastructure Configuration:

• Refined monitoring configuration in main.bicep by removing redundant diagnostic settings assignment, streamlining resource deployment logic.

These changes collectively improve the reliability, maintainability, and observability of the codebase.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

[Copilot]

Pull request overview

This PR improves observability across the FastAPI backend by centralizing Application Insights/OpenTelemetry setup, adding richer request/service logging + custom event telemetry, and standardizing deployment parameters/scripts in the infra layer.

Changes:

• Centralizes Azure Monitor/OpenTelemetry configuration in src/api/app.py and adds span processors to reduce telemetry noise.
• Adds structured logging + Application Insights custom event tracking across chat/history endpoints and services.
• Standardizes infra parameter names and improves deployment output parsing robustness; updates default jumpbox VM size to Standard_D2s_v5.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/tests/api/common/logging/test_event_utils.py	Updates test expectations for new warning message + logger scoping.
src/api/services/history_service.py	Adds structured info/exception logging around conversation history operations.
src/api/services/chat_service.py	Adds structured logging and emits completion telemetry for chat streams.
src/api/helpers/chat_helper.py	Converts log statements to structured logging format.
src/api/common/logging/span_filters.py	Adds custom span processors intended to reduce noisy spans in App Insights.
src/api/common/logging/event_utils.py	Removes stdout printing and standardizes warning logging for telemetry gating.
src/api/app.py	Centralizes Azure Monitor configuration + FastAPI instrumentation and suppresses noisy SDK loggers.
src/api/api/history_routes.py	Removes per-module AI config and adds structured logs + error/success telemetry across history endpoints.
src/api/api/api_routes.py	Removes per-module AI config; adds structured logs + chat request/user correlation + error telemetry.
infra/scripts/run_create_agents_scripts.sh	Makes deployment output parsing case-insensitive.
infra/scripts/process_sample_data.sh	Makes deployment output parsing case-insensitive.
infra/scripts/process_custom_data.sh	Makes deployment output parsing case-insensitive.
infra/modules/virtualNetwork.bicep	Updates VM sizing guidance comments.
infra/main_custom.bicep	Changes default jumpbox VM size to Standard_D2s_v5.
infra/main.waf.parameters.json	Renames standardized parameters; adds vmSize parameter plumbing.
infra/main.parameters.json	Renames standardized parameters for deployment type/capacity.
infra/main.json	Regenerated ARM template reflecting VM size default + minor ordering changes.
infra/main.bicep	Removes redundant diagnostics assignment and updates default jumpbox VM size.
documents/TroubleShootingSteps.md	Expands SKU-not-available guidance and documents the new default VM size.
documents/CustomizingAzdParameters.md	Documents new AZURE_ENV_VM_SIZE parameter.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.