Update Friendly Evaluator#46076
Open
w-javed wants to merge 17 commits intofeature/azure-ai-projects/2.0.2from
Open
Update Friendly Evaluator#46076w-javed wants to merge 17 commits intofeature/azure-ai-projects/2.0.2from
w-javed wants to merge 17 commits intofeature/azure-ai-projects/2.0.2from
Conversation
w-javed
requested review from
bobogogo1990,
dargilco,
glharper,
howieleung,
kingernupur,
nick863,
trangevi and
trrwilson
as code owners
API Change CheckAPIView identified API level changes in this PR and created the following API reviews |
YoYoJa
approved these changes
Apr 2, 2026
aprilk-ms
reviewed
Apr 2, 2026
...ure-ai-projects/samples/evaluations/custom_evaluators/friendly_evaluator/common_util/util.py
Outdated
Show resolved
Hide resolved
aprilk-ms
reviewed
Apr 2, 2026
...e-ai-projects/samples/evaluations/custom_evaluators/friendly_evaluator/friendly_evaluator.py
Outdated
Show resolved
Hide resolved
aprilk-ms
reviewed
Apr 2, 2026
...e-ai-projects/samples/evaluations/custom_evaluators/friendly_evaluator/friendly_evaluator.py
Outdated
Show resolved
Hide resolved
aprilk-ms
reviewed
Apr 2, 2026
...e-ai-projects/samples/evaluations/custom_evaluators/friendly_evaluator/friendly_evaluator.py
Outdated
Show resolved
Hide resolved
aprilk-ms
reviewed
Apr 2, 2026
...ure-ai-projects/samples/evaluations/custom_evaluators/friendly_evaluator/common_util/util.py
Show resolved
Hide resolved
w-javed
requested review from
a team,
ChenxiJiang333,
annatisch,
benbp,
catalinaperalta,
iscai-msft,
l0lawrence,
lmazuel,
maririos,
mccoyp,
msyyc,
praveenkuttappan,
scbedd and
weshaggard
as code owners
w-javed
requested review from
johanste,
kashifkhan,
pvaneck and
xiangyan99
as code owners
Update the FriendlyEvaluator sample to return the new standard output format with score, label, reason, threshold, and passed at the top level. Extra evaluator output fields (explanation, tone, confidence) are nested under a properties dict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use 'from openai import OpenAI' instead of AzureOpenAI - Accept api_key and model params instead of model_config dict - Use client.responses.create() instead of chat.completions.create() - Update util.py: split build_evaluation_messages into build_evaluation_instructions() and build_evaluation_input() - Update sample init_parameters schema accordingly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address aprilk-ms review: annotate which fields in the evaluation result dict are required vs optional for the evaluation service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Align sample_eval_upload_friendly_evaluator.py with the updated FriendlyEvaluator that takes api_key and model instead of deployment_name/model_config. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
w-javed
force-pushed
the
waqasjaved02/friendly-evaluator-properties-output
branch
from
5c23650 to
805a66c
Compare
Merge sample_custom_evaluator_friendly_evaluator.py into sample_eval_upload_friendly_evaluator.py so the sample first runs FriendlyEvaluator locally, then uploads, creates eval, and runs it. Fix model_name parameter to match evaluator __init__ signature. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
|
Reminder to run 'black' tool. Thanks! |
Member
|
Regarding the MyPy error, note that Azure SDK has recently updated their tools. They no longer use "tox". This is the new command to run MyPy: |
dargilco
approved these changes
Apr 3, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aprilk-ms
reviewed
Apr 3, 2026
...cts/samples/evaluations/custom_evaluators/answer_length_evaluator/answer_length_evaluator.py
Outdated
Show resolved
Hide resolved
aprilk-ms
reviewed
Apr 3, 2026
...ure-ai-projects/samples/evaluations/custom_evaluators/friendly_evaluator/common_util/util.py
Outdated
Show resolved
Hide resolved
aprilk-ms
reviewed
Apr 3, 2026
sdk/ai/azure-ai-projects/samples/evaluations/sample_custom_eval_upload_advanced.py
Show resolved
Hide resolved
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Rename sample files: simple (AnswerLength) and advanced (Friendly) - Remove config/threshold from AnswerLengthEvaluator, add label output - Remove passed from FriendlyEvaluator, keep threshold - Update README Evaluator Types section with new samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aprilk-ms
reviewed
Apr 4, 2026
sdk/ai/azure-ai-projects/samples/evaluations/sample_custom_eval_upload_simple.py
Outdated
Show resolved
Hide resolved
…l_upload_simple.py Co-authored-by: aprilk-ms <55356546+aprilk-ms@users.noreply.github.com>
aprilk-ms
reviewed
Apr 4, 2026
...cts/samples/evaluations/custom_evaluators/answer_length_evaluator/answer_length_evaluator.py
Outdated
Show resolved
Hide resolved
…/answer_length_evaluator/answer_length_evaluator.py Co-authored-by: aprilk-ms <55356546+aprilk-ms@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.