评估将结构化反馈、分数或基本事实附加到 MLflow 中质量评估和改进的跟踪和范围。
MLflow 提供两个 API：
-
mlflow.log_feedback()- 记录评估应用的实际输出或中间步骤的 反馈 （例如“响应是否良好？”、评分、评论） -
mlflow.log_assessment()- 适用于任何评估类型的通用 API，包括定义应用程序应生成的理想或正确结果（基准事实）的期望值
本参考提供了有关如何使用这些 API 的综合示例。
有关反馈和期望数据模型的更多详细信息，请参阅 跟踪数据模型 。
例子
运行以下示例将会生成一个跟踪记录，如下所示。
import mlflow
from mlflow.entities.assessment import (
AssessmentSource,
AssessmentSourceType,
AssessmentError,
)
@mlflow.trace
def my_app(input: str) -> str:
return input + "_output"
# Create a sample trace to demonstrate assessment logging
my_app(input="hello")
trace_id = mlflow.get_last_active_trace_id()
# Handle case where trace_id might be None
if trace_id is None:
raise ValueError("No active trace found. Make sure to run a traced function first.")
print(f"Using trace_id: {trace_id}")
# =============================================================================
# LOG_FEEDBACK - Evaluating actual outputs and performance
# =============================================================================
# Example 1: Human rating (integer scale)
# Use case: Domain experts rating response quality on a 1-5 scale
mlflow.log_feedback(
trace_id=trace_id,
name="human_rating",
value=4, # int - rating scale feedback
rationale="Human evaluator rating",
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="evaluator@company.com",
),
)
# Example 2: LLM judge score (float for precise scoring)
# Use case: Automated quality assessment using LLM-as-a-judge
mlflow.log_feedback(
trace_id=trace_id,
name="llm_judge_score",
value=0.85, # float - precise scoring from 0.0 to 1.0
rationale="LLM judge evaluation",
source=AssessmentSource(
source_type=AssessmentSourceType.LLM_JUDGE,
source_id="gpt-4o-mini",
),
metadata={"temperature": "0.1", "model_version": "2024-01"},
)
# Example 3: Binary feedback (boolean for yes/no assessments)
# Use case: Simple thumbs up/down or correct/incorrect evaluations
mlflow.log_feedback(
trace_id=trace_id,
name="is_helpful",
value=True, # bool - binary assessment
rationale="Boolean assessment of helpfulness",
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="reviewer@company.com",
),
)
# Example 4: Multi-category feedback (list for multiple classifications)
# Use case: Automated categorization or multi-label classification
mlflow.log_feedback(
trace_id=trace_id,
name="automated_categories",
value=["helpful", "accurate", "concise"], # list - multiple categories
rationale="Automated categorization",
source=AssessmentSource(
source_type=AssessmentSourceType.CODE,
source_id="classifier_v1.2",
),
)
# Example 5: Complex analysis with metadata (when you need structured context)
# Use case: Detailed automated analysis with multiple dimensions stored in metadata
mlflow.log_feedback(
trace_id=trace_id,
name="response_analysis_score",
value=4.2, # single score instead of dict - keeps value simple
rationale="Analysis: 150 words, positive sentiment, includes examples, confidence 0.92",
source=AssessmentSource(
source_type=AssessmentSourceType.CODE,
source_id="analyzer_v2.1",
),
metadata={ # Use metadata for structured details
"word_count": "150",
"sentiment": "positive",
"has_examples": "true",
"confidence": "0.92",
},
)
# Example 6: Error handling when evaluation fails
# Use case: Logging when automated evaluators fail due to API limits, timeouts, etc.
mlflow.log_feedback(
trace_id=trace_id,
name="failed_evaluation",
source=AssessmentSource(
source_type=AssessmentSourceType.LLM_JUDGE,
source_id="gpt-4o",
),
error=AssessmentError( # Use error field when evaluation fails
error_code="RATE_LIMIT_EXCEEDED",
error_message="API rate limit exceeded during evaluation",
),
metadata={"retry_count": "3", "error_timestamp": "2024-01-15T10:30:00Z"},
)
# =============================================================================
# LOG_EXPECTATION - Defining ground truth and desired outcomes
# =============================================================================
# Example 1: Simple text expectation (most common pattern)
# Use case: Defining the ideal response for factual questions
mlflow.log_expectation(
trace_id=trace_id,
name="expected_response",
value="The capital of France is Paris.", # Simple string - the "correct" answer
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="content_curator@example.com",
),
)
# Example 2: Complex structured expectation (advanced pattern)
# Use case: Defining detailed requirements for response structure and content
mlflow.log_expectation(
trace_id=trace_id,
name="expected_response_structure",
value={ # Complex dict - detailed specification of ideal response
"entities": {
"people": ["Marie Curie", "Pierre Curie"],
"locations": ["Paris", "France"],
"dates": ["1867", "1934"],
},
"key_facts": [
"First woman to win Nobel Prize",
"Won Nobel Prizes in Physics and Chemistry",
"Discovered radium and polonium",
],
"response_requirements": {
"tone": "informative",
"length_range": {"min": 100, "max": 300},
"include_examples": True,
"citations_required": False,
},
},
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="content_strategist@example.com",
),
metadata={
"content_type": "biographical_summary",
"target_audience": "general_public",
"fact_check_date": "2024-01-15",
},
)
# Example 3: Multiple acceptable answers (list pattern)
# Use case: When there are several valid ways to express the same fact
mlflow.log_expectation(
trace_id=trace_id,
name="expected_facts",
value=[ # List of acceptable variations of the correct answer
"Paris is the capital of France",
"The capital city of France is Paris",
"France's capital is Paris",
],
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="qa_team@example.com",
),
)