How to Configure Open AI Code Review in Azure DevOps
AI Code Review Setup in Azure DevOps
This guide summarizes the steps to configure AI code review for another Azure DevOps project using the Open AI Code Review extension.
1. Install the extension
Go to:
Organization Settings → Extensions → Browse marketplace
Install:
Open AI Code Review
Confirm your organization approves use of this third-party extension.
2. Create or open the pipeline
Go to:
Pipelines → New pipeline
Select:
Azure Repos Git → choose the repository
You can use either YAML or Classic editor. If a pipeline already exists, open it and edit it instead of creating a new one.
3. Add the AI review task
If using Classic editor, go to:
Pipeline → Tasks → Agent job 1 → +
Search for:
Open AI Code Review
Add the task.
4. Enable OAuth token access
In the pipeline editor, select:
Agent job 1 → Additional options
Enable:
Allow scripts to access the OAuth token
This is important if the extension needs to interact with pull requests.
5. Store the OpenAI API key securely
Open the pipeline Variables section and create a secret variable, for example:
OpenAI_ApiKey
Then in the task, set:
Open AI API Key → $(OpenAI_ApiKey)
This is safer than pasting the API key directly into the task field.
6. Configure the AI task
Fill the task fields:
- Open AI API Key →
$(OpenAI_ApiKey) - Open AI API Model →
gpt-4(recommended for the current extension version if available in the dropdown) - Check for bugs → enabled
- Check for performance problems → enabled
- Check for missed best practices → enabled
- File Extensions → for example:
cs,js,ts,php,json,sql,xml,config - Files to exclude → optional
- Additional Prompts → optional, for example:
Review for suspicious code, misspellings, performance issues, incorrect logic, dead code, confusing naming, and security issues. Be concise and specific. If no issues are found, say so clearly.
7. Save the pipeline
Click:
Save
You can run it once manually to confirm the task is configured correctly.
8. Grant repository permission to the build service
Go to:
Repos → Repositories → [Repository Name] → Security
Find and select the build service user, usually something like:
[ProjectName] Build Service ([CollectionName])
Set this permission to:
Contribute to pull requests → Allow
9. Add branch policy for PR validation
Go to:
Repos → Branches → main (or master) → ... → Branch policies
Under Build Validation, click:
+ Add build policy
Select:
- Pipeline → your AI review pipeline
- Trigger → Automatic
- Required → enabled if you want the PR blocked until validation completes
Save the policy.
10. Optional: enable CI trigger
If you also want the pipeline to run on normal branch pushes, go to:
Triggers
Enable:
- Continuous integration → ON
- Branch filter → usually main or master
This is optional and separate from PR validation.
11. Test with a real code PR
Create or update a pull request with changes in supported file types, for example:
.cs.php.js.ts.json
Avoid testing only with unsupported files unless you added those file types in the task’s file extensions.
Expected Result
- When a pull request is created or updated, the build starts automatically through branch policy validation.
- The AI review task analyzes the changed files.
- The PR waits for build validation before merge if the policy is required.
- If the extension supports it and permissions are correct, AI feedback can appear in the PR.
Common Issues
- No AI feedback → changed file type is not included in File Extensions.
- No PR comments → Allow scripts to access the OAuth token is not enabled, or the build service is missing Contribute to pull requests permission.
- Pipeline runs manually only → branch policy or trigger is not configured correctly.
- Build succeeds but no review output → no supported code files were changed.
- API key is exposed in the task → move it to a secret pipeline variable.
Recommended File Extensions
Adjust these per project. Example:
cs,js,ts,php,json,sql,xml,config