Testing the Course Samples

Validate that the lesson notebooks and code samples run against a live

Microsoft Foundry / Azure OpenAI setup. The repo ships a runner at

[scripts/validate-notebooks.ps1](../../../scripts/validate-notebooks.ps1) that

executes every Python notebook headlessly and prints a PASS/FAIL matrix.

When to use

• "Validate all the notebooks / samples against my Azure subscription."
• "Smoke-test the course after upgrading packages or changing models."
• "Which lessons still pass / fail live?"

Do not use this for the AI Smoke Test GitHub Action (that validates deployed

hosted agents — see [tests/README.md](../../../tests/README.md)). This skill

runs the notebooks locally.

Prerequisites (check first)

1. Python 3.12+ with course deps: python -m pip install -r requirements.txt

plus the executor: python -m pip install nbconvert ipykernel.

1. .env at the repo root (copy from [.env.example](../../../.env.example)) with at least:

• AZURE_AI_PROJECT_ENDPOINT — Foundry project endpoint

(https://<account>.services.ai.azure.com/api/projects/<project>)

• AZURE_AI_MODEL_DEPLOYMENT_NAME — a non-deprecated deployment (e.g. gpt-5-mini)
• AZURE_OPENAI_ENDPOINT (https://<account>.openai.azure.com) and AZURE_OPENAI_DEPLOYMENT

for lessons that call Azure OpenAI directly (Lesson 06, 02-azure-openai, 14 handoff/human-loop).

1. az login completed — samples authenticate with AzureCliCredential (Entra ID, keyless).
2. Verify the model deployment exists:

az cognitiveservices account deployment list -g <rg> -n <account> -o table.

Running the validation

# All Python notebooks (skips .NET, .venv, site-packages, translations, skill assets)
pwsh scripts/validate-notebooks.ps1

# A single lesson, with a longer per-cell timeout
pwsh scripts/validate-notebooks.ps1 -Filter '08-*' -Timeout 600

# Just list what would run (no execution)
pwsh scripts/validate-notebooks.ps1 -List

# Explicit interpreter (if `python` is not on PATH, e.g. Windows Store alias)
pwsh scripts/validate-notebooks.ps1 -Python "C:/path/to/python.exe"

The script writes executed copies, per-notebook logs, and results.json to

$env:TEMP\aiab-nbval and exits with the number of failures.

Transient failures (shared-subscription HTTP 429 rate limits, an occasional

AzureCliCredential token hiccup, or a timeout) are retried automatically

(-Retries, default 2, with -RetryDelaySeconds backoff, default 20). If a

model deployment is regularly 429-ing, check the subscription's GlobalStandard

TPM quota (az cognitiveservices usage list -l <region>) — raising a single

deployment's capacity does not help when the subscription quota is exhausted.

Interpreting results

• PASS — the notebook ran end-to-end with no cell error.
• FAIL — the first Error / Exception line is shown; open the matching

log_*.txt in the output dir for the full traceback.

• A single notebook's failure is bounded by -Timeout (per cell), so a hung

human-in-the-loop cell surfaces as StdinNotImplementedError rather than hanging.

Lessons that need extra resources (expected to fail without them)

Reporting back

Summarise as a PASS/FAIL table grouped by lesson. Separate genuine regressions

(code/config bugs to fix) from environment gaps (missing Search/Foundry Local/PAT),

and cite the failing log_*.txt for each real failure.