評価は、MLflow の品質評価と改善のために、トレースやスパンに構造化されたフィードバック、スコア、またはグラウンド トゥルースを付与します。
MLflow には、次の 2 つの API が用意されています。
-
mlflow.log_feedback()- アプリの実際の出力または中間ステップを評価する フィードバック をログに記録します (例: "応答は良好でしたか?"、評価、コメント) -
mlflow.log_assessment()- あらゆる評価の種類に対応する一般的な API、アプリが生成する必要がある望ましい結果または正しい結果 (基準値) を定義する 期待 値を含む
このリファレンスでは、これらの API を使用する方法の包括的な例を示します。
フィードバックおよび期待値データモデルの詳細については、トレースデータモデルをご参照ください。
例示
次の例を実行すると、次に示すようにトレースが生成されます。
import mlflow
from mlflow.entities.assessment import (
AssessmentSource,
AssessmentSourceType,
AssessmentError,
)
@mlflow.trace
def my_app(input: str) -> str:
return input + "_output"
# Create a sample trace to demonstrate assessment logging
my_app(input="hello")
trace_id = mlflow.get_last_active_trace_id()
# Handle case where trace_id might be None
if trace_id is None:
raise ValueError("No active trace found. Make sure to run a traced function first.")
print(f"Using trace_id: {trace_id}")
# =============================================================================
# LOG_FEEDBACK - Evaluating actual outputs and performance
# =============================================================================
# Example 1: Human rating (integer scale)
# Use case: Domain experts rating response quality on a 1-5 scale
mlflow.log_feedback(
trace_id=trace_id,
name="human_rating",
value=4, # int - rating scale feedback
rationale="Human evaluator rating",
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="evaluator@company.com",
),
)
# Example 2: LLM judge score (float for precise scoring)
# Use case: Automated quality assessment using LLM-as-a-judge
mlflow.log_feedback(
trace_id=trace_id,
name="llm_judge_score",
value=0.85, # float - precise scoring from 0.0 to 1.0
rationale="LLM judge evaluation",
source=AssessmentSource(
source_type=AssessmentSourceType.LLM_JUDGE,
source_id="gpt-4o-mini",
),
metadata={"temperature": "0.1", "model_version": "2024-01"},
)
# Example 3: Binary feedback (boolean for yes/no assessments)
# Use case: Simple thumbs up/down or correct/incorrect evaluations
mlflow.log_feedback(
trace_id=trace_id,
name="is_helpful",
value=True, # bool - binary assessment
rationale="Boolean assessment of helpfulness",
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="reviewer@company.com",
),
)
# Example 4: Multi-category feedback (list for multiple classifications)
# Use case: Automated categorization or multi-label classification
mlflow.log_feedback(
trace_id=trace_id,
name="automated_categories",
value=["helpful", "accurate", "concise"], # list - multiple categories
rationale="Automated categorization",
source=AssessmentSource(
source_type=AssessmentSourceType.CODE,
source_id="classifier_v1.2",
),
)
# Example 5: Complex analysis with metadata (when you need structured context)
# Use case: Detailed automated analysis with multiple dimensions stored in metadata
mlflow.log_feedback(
trace_id=trace_id,
name="response_analysis_score",
value=4.2, # single score instead of dict - keeps value simple
rationale="Analysis: 150 words, positive sentiment, includes examples, confidence 0.92",
source=AssessmentSource(
source_type=AssessmentSourceType.CODE,
source_id="analyzer_v2.1",
),
metadata={ # Use metadata for structured details
"word_count": "150",
"sentiment": "positive",
"has_examples": "true",
"confidence": "0.92",
},
)
# Example 6: Error handling when evaluation fails
# Use case: Logging when automated evaluators fail due to API limits, timeouts, etc.
mlflow.log_feedback(
trace_id=trace_id,
name="failed_evaluation",
source=AssessmentSource(
source_type=AssessmentSourceType.LLM_JUDGE,
source_id="gpt-4o",
),
error=AssessmentError( # Use error field when evaluation fails
error_code="RATE_LIMIT_EXCEEDED",
error_message="API rate limit exceeded during evaluation",
),
metadata={"retry_count": "3", "error_timestamp": "2024-01-15T10:30:00Z"},
)
# =============================================================================
# LOG_EXPECTATION - Defining ground truth and desired outcomes
# =============================================================================
# Example 1: Simple text expectation (most common pattern)
# Use case: Defining the ideal response for factual questions
mlflow.log_expectation(
trace_id=trace_id,
name="expected_response",
value="The capital of France is Paris.", # Simple string - the "correct" answer
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="content_curator@example.com",
),
)
# Example 2: Complex structured expectation (advanced pattern)
# Use case: Defining detailed requirements for response structure and content
mlflow.log_expectation(
trace_id=trace_id,
name="expected_response_structure",
value={ # Complex dict - detailed specification of ideal response
"entities": {
"people": ["Marie Curie", "Pierre Curie"],
"locations": ["Paris", "France"],
"dates": ["1867", "1934"],
},
"key_facts": [
"First woman to win Nobel Prize",
"Won Nobel Prizes in Physics and Chemistry",
"Discovered radium and polonium",
],
"response_requirements": {
"tone": "informative",
"length_range": {"min": 100, "max": 300},
"include_examples": True,
"citations_required": False,
},
},
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="content_strategist@example.com",
),
metadata={
"content_type": "biographical_summary",
"target_audience": "general_public",
"fact_check_date": "2024-01-15",
},
)
# Example 3: Multiple acceptable answers (list pattern)
# Use case: When there are several valid ways to express the same fact
mlflow.log_expectation(
trace_id=trace_id,
name="expected_facts",
value=[ # List of acceptable variations of the correct answer
"Paris is the capital of France",
"The capital city of France is Paris",
"France's capital is Paris",
],
source=AssessmentSource(
source_type=AssessmentSourceType.HUMAN,
source_id="qa_team@example.com",
),
)
次のステップ
- ユーザー フィードバックの収集 - 評価ログを適用して運用フィードバックを収集する
- 既存のトレースにラベルを付 ける - レビュー アプリを使用して体系的に評価を追加する
- 評価データセットの構築 - 期待される評価を評価データセットに変換する