Production tracing for Azure AI Foundry workloads

Why prototype instrumentation fails in production

Console logging around a single API call does not survive multi-step flows: retrieval, reranking, tool calls, guardrails, and downstream HTTP dependencies each add latency and failure modes.

Production teams need request-scoped correlation across every step. That means a trace ID propagated from your API boundary through Foundry-orchestrated flows to model endpoints and back.

Industry pattern: treat each user-facing AI action as a transaction with SLIs — availability, latency, error rate, and quality regressions — not as an isolated completion call.

Minimum viable signals (what to capture)

Identity and versioning: model deployment name, prompt or flow version, environment (dev/staging/prod), and caller service.

Performance: end-to-end latency, time-to-first-token where streaming applies, and per-step duration inside the flow.

Cost: input/output token counts and compute SKU where self-hosted inference applies — attach as trace dimensions for chargeback dashboards.

Outcome: success/failure, filtered/blocked by content safety, timeout, and downstream dependency status codes.

Context for triage: hashed or redacted user/session identifiers where privacy policy allows — enough to correlate without logging sensitive prompt content verbatim.

Application Insights and Foundry observability

Application Insights (or OpenTelemetry export into Azure Monitor) is the durable store most Azure platform teams already operate. Foundry and Azure OpenAI emit telemetry that should land in the same workspace your SRE team pages on.

Structure spans so operators can pivot: outer span for the HTTP/API request, child spans for retrieval, flow steps, model inference, and post-processing. When an incident starts, the question is user impact — not which log file to grep.

Dashboards worth building early: p95 latency by model version, error rate by deployment, token volume by product area, and content-filter block rate. Alert on SLO burn and dependency degradation, not raw token volume alone.

RAG and agent flows need explicit step boundaries

Retrieval-augmented generation introduces silent failure: empty retrieval, stale index, wrong embedding model, or high-latency vector search. Each step deserves its own span with document count and retrieval latency attached.

Agent flows that call tools or external APIs multiply blast radius. Record tool name, latency, and result status — without storing secrets or PII in span attributes.

Evaluation hooks belong beside tracing: sample production traces into offline eval sets, compare prompt revisions before promotion, and block production promotion when regression thresholds fail.

Governance tied to telemetry

Tracing without governance becomes expensive noise. Pair telemetry with RBAC on Foundry workspaces, private endpoints for inference paths, token budgets with alerts, and CI/CD gates that require a passing eval or staging soak before prod.

Rollback should be operable: if a new prompt revision increases error rate or latency beyond error budget, revert to the last known-good deployment using the same promotion path — not a manual key swap.

Where BetterCallDevOps fits

We help teams stand up the production baseline: Foundry workspace layout, model deployment patterns, tracing integration, cost dashboards, and promotion runbooks your engineers can own.

Engagements start with a readiness review (typically 3–10 days) and move into fixed-scope implementation when the backlog is clear. Anonymized delivery patterns are documented in our case studies and reference architectures on this site.