单目3D初始代码
This commit is contained in:
92
eval_tools/model_comparison/README_per_case_comparison.md
Executable file
92
eval_tools/model_comparison/README_per_case_comparison.md
Executable file
@@ -0,0 +1,92 @@
|
||||
# Per-Case 2D Metrics Comparison Tool
|
||||
|
||||
This tool compares `per_case_2d` metrics between two model evaluation reports and identifies cases with significant metric differences.
|
||||
|
||||
## Files
|
||||
|
||||
- `compare_per_case_2d.py` - Main Python script for comparing per-case metrics
|
||||
- `compare_per_case_2d.sh` - Shell script with pre-configured paths for mono3d vs yolov5s-300w-newdata comparison
|
||||
|
||||
## Usage
|
||||
|
||||
### Quick Start (Using Shell Script)
|
||||
|
||||
```bash
|
||||
cd /deeplearning_team/ydong/dongying/projects/yolov5-3d
|
||||
./eval_tools/model_comparison/compare_per_case_2d.sh
|
||||
```
|
||||
|
||||
This will compare the two models and save results to `evaluation_results/per_case_2d_comparison.json`.
|
||||
|
||||
### Custom Comparison (Using Python Script)
|
||||
|
||||
```bash
|
||||
python eval_tools/model_comparison/compare_per_case_2d.py \
|
||||
--model1 path/to/model1/evaluation_report.json \
|
||||
--model2 path/to/model2/evaluation_report.json \
|
||||
--model1-name "Model-A" \
|
||||
--model2-name "Model-B" \
|
||||
--threshold 0.1 \
|
||||
--output comparison_results.json \
|
||||
--top-n 30
|
||||
```
|
||||
|
||||
### Arguments
|
||||
|
||||
- `--model1`: Path to first model's evaluation_report.json (required)
|
||||
- `--model2`: Path to second model's evaluation_report.json (required)
|
||||
- `--model1-name`: Display name for model 1 (default: "Model-1")
|
||||
- `--model2-name`: Display name for model 2 (default: "Model-2")
|
||||
- `--threshold`: Threshold for significant difference, e.g., 0.1 = 10% (default: 0.1)
|
||||
- `--output`: Output JSON file path (default: "per_case_comparison.json")
|
||||
- `--top-n`: Number of top different cases to display (default: 20)
|
||||
|
||||
## Output
|
||||
|
||||
The script generates:
|
||||
|
||||
1. **Console Output**:
|
||||
- Summary of total cases and common cases
|
||||
- Top N cases with significant differences
|
||||
- Summary statistics (mean, std, median, range) for each class and metric
|
||||
|
||||
2. **JSON File**: Contains detailed comparison data including:
|
||||
- `summary`: Overview statistics
|
||||
- `significant_differences`: List of cases exceeding the threshold
|
||||
- `all_case_comparisons`: Complete per-case comparison data
|
||||
- `summary_statistics`: Statistical analysis by class and metric
|
||||
|
||||
## Example Output
|
||||
|
||||
```
|
||||
Top 30 Cases with Significant Differences
|
||||
================================================================================
|
||||
|
||||
1. Case: 20251118/seq-53
|
||||
Class: pedestrian, Metric: ap
|
||||
mono3d: 1.0000
|
||||
yolov5s-300w-newdata: 0.0000
|
||||
Difference: -1.0000 (abs: 1.0000)
|
||||
|
||||
2. Case: 20251121/seq-30
|
||||
Class: roadblock, Metric: ap
|
||||
mono3d: 1.0000
|
||||
yolov5s-300w-newdata: 0.0000
|
||||
Difference: -1.0000 (abs: 1.0000)
|
||||
...
|
||||
|
||||
Summary Statistics
|
||||
================================================================================
|
||||
|
||||
VEHICLE:
|
||||
ap : mean=-0.0776, std=0.1439, median=-0.0243, range=[-0.7935, +0.0994]
|
||||
precision : mean=+0.1279, std=0.2248, median=+0.0934, range=[-0.9442, +0.6074]
|
||||
recall : mean=-0.1210, std=0.1579, median=-0.0635, range=[-0.8975, +0.0000]
|
||||
```
|
||||
|
||||
## Interpretation
|
||||
|
||||
- **Positive difference**: Model 2 performs better than Model 1
|
||||
- **Negative difference**: Model 1 performs better than Model 2
|
||||
- Cases are sorted by absolute difference (largest differences first)
|
||||
- Summary statistics show overall trends across all cases
|
||||
3
eval_tools/model_comparison/bad_eval_cases.txt
Executable file
3
eval_tools/model_comparison/bad_eval_cases.txt
Executable file
@@ -0,0 +1,3 @@
|
||||
019b8ddb-ae8a-70f3-b86f-055894c79724
|
||||
019b6bf2-01a6-7029-9232-fce2bbcd2d73
|
||||
019b6bf2-0124-7820-91b7-c5eb42150cd2
|
||||
1045
eval_tools/model_comparison/compare_models.py
Executable file
1045
eval_tools/model_comparison/compare_models.py
Executable file
File diff suppressed because it is too large
Load Diff
82
eval_tools/model_comparison/compare_models_example.sh
Executable file
82
eval_tools/model_comparison/compare_models_example.sh
Executable file
@@ -0,0 +1,82 @@
|
||||
#!/bin/bash
|
||||
# Example script for comparing two model evaluation results
|
||||
#
|
||||
# Usage: bash eval_tools/compare_models_example.sh
|
||||
|
||||
# =============================================================================
|
||||
# Configuration
|
||||
# =============================================================================
|
||||
|
||||
# Model 1 (mono3d)
|
||||
|
||||
# Set PYTHONPATH to project root for module imports
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
|
||||
MODEL1_REPORT="eval_results_multiprocess/mono3d/20260203_162537/evaluation_report.json"
|
||||
|
||||
MODEL1_NAME="mono3d"
|
||||
|
||||
# Model 2 (yolov5s-300w)
|
||||
MODEL2_REPORT="eval_results_multiprocess/yolov5s/20260203_161644/evaluation_report.json"
|
||||
MODEL2_NAME="yolov5s-300w"
|
||||
|
||||
# Output directory
|
||||
OUTPUT_DIR="comparison_results/$(date +%Y%m%d_%H%M%S)"
|
||||
|
||||
# =============================================================================
|
||||
# Run Comparison
|
||||
# =============================================================================
|
||||
|
||||
echo "=========================================="
|
||||
echo "Model Evaluation Comparison"
|
||||
echo "=========================================="
|
||||
echo "Model 1: $MODEL1_NAME"
|
||||
echo " Report: $MODEL1_REPORT"
|
||||
echo "Model 2: $MODEL2_NAME"
|
||||
echo " Report: $MODEL2_REPORT"
|
||||
echo "Output: $OUTPUT_DIR"
|
||||
echo "=========================================="
|
||||
|
||||
# Check if reports exist
|
||||
if [ ! -f "$MODEL1_REPORT" ]; then
|
||||
echo "Error: Model 1 report not found: $MODEL1_REPORT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$MODEL2_REPORT" ]; then
|
||||
echo "Error: Model 2 report not found: $MODEL2_REPORT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Run comparison with visualization
|
||||
python eval_tools/model_comparison/compare_models_visualize.py \
|
||||
--model1 "$MODEL1_REPORT" \
|
||||
--model2 "$MODEL2_REPORT" \
|
||||
--output-dir "$OUTPUT_DIR" \
|
||||
--model1-name "$MODEL1_NAME" \
|
||||
--model2-name "$MODEL2_NAME"
|
||||
|
||||
# Check if comparison was successful
|
||||
if [ $? -eq 0 ]; then
|
||||
echo ""
|
||||
echo "✓ Comparison completed successfully!"
|
||||
echo ""
|
||||
echo "View results:"
|
||||
echo " Text report: $OUTPUT_DIR/comparison_report.txt"
|
||||
echo " JSON report: $OUTPUT_DIR/comparison_report.json"
|
||||
echo " Plots: $OUTPUT_DIR/comparison_*.png"
|
||||
echo ""
|
||||
|
||||
# Display summary from text report
|
||||
if [ -f "$OUTPUT_DIR/comparison_report.txt" ]; then
|
||||
echo "=========================================="
|
||||
echo "Quick Summary (from report):"
|
||||
echo "=========================================="
|
||||
grep -A 20 "2D DETECTION METRICS - OVERALL COMPARISON" "$OUTPUT_DIR/comparison_report.txt" | head -25
|
||||
fi
|
||||
else
|
||||
echo ""
|
||||
echo "✗ Comparison failed!"
|
||||
exit 1
|
||||
fi
|
||||
48
eval_tools/model_comparison/compare_models_only.sh
Executable file
48
eval_tools/model_comparison/compare_models_only.sh
Executable file
@@ -0,0 +1,48 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Compare Models Only (Skip Evaluation Steps)
|
||||
#
|
||||
# This script runs only the comparison steps (Step 4-5), assuming that
|
||||
# evaluation and common match finding have already been completed.
|
||||
#
|
||||
# Usage:
|
||||
# bash eval_tools/compare_models_only.sh <MODEL1_DIR> <MODEL2_DIR> <COMMON_MATCHES_JSON>
|
||||
#
|
||||
# Example:
|
||||
# bash eval_tools/compare_models_only.sh \
|
||||
# eval_results_common_match_comparison/mono3d/20260203_210259 \
|
||||
# eval_results_common_match_comparison/yolov5s-300w/20260203_210259 \
|
||||
# eval_results_common_match_comparison/common_matches_20260203_210259/common_matches.json
|
||||
|
||||
|
||||
set -e # Exit on error
|
||||
|
||||
# Set PYTHONPATH to project root for module imports
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
|
||||
|
||||
MODEL1_DIR="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/yolov5s-300w-newdata/20260228_102849"
|
||||
MODEL2_DIR="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/yolov5s-300w-newdata-cncap/20260228_102849"
|
||||
MODEL1_NAME="yolov5s-300w-newdata"
|
||||
MODEL2_NAME="yolov5s-300w-newdata-cncap"
|
||||
COMMON_MATCHES_DIR="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/common_matches_20260228_102849"
|
||||
COMPARISON_DIR="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/comparison_common_matches_20260228_102849-v2"
|
||||
|
||||
python eval_tools/model_comparison/compare_models.py \
|
||||
--model1 ${MODEL1_DIR}/evaluation_report.json \
|
||||
--model2 ${MODEL2_DIR}/evaluation_report.json \
|
||||
--model1-name "${MODEL1_NAME}" \
|
||||
--model2-name "${MODEL2_NAME}" \
|
||||
--common-matches ${COMMON_MATCHES_DIR}/common_matches.json \
|
||||
--output-dir ${COMPARISON_DIR}
|
||||
|
||||
echo "✓ Comparison results saved to: ${COMPARISON_DIR}"
|
||||
|
||||
|
||||
# python eval_tools/model_comparison/compare_models.py \
|
||||
# --model1 ${MODEL1_DIR}/evaluation_report.json \
|
||||
# --model2 ${MODEL2_DIR}/evaluation_report.json \
|
||||
# --model1-name "${MODEL1_NAME}" \
|
||||
# --model2-name "${MODEL2_NAME}" \
|
||||
# --output-dir ${COMPARISON_TRADITIONAL_DIR}
|
||||
374
eval_tools/model_comparison/compare_models_visualize.py
Executable file
374
eval_tools/model_comparison/compare_models_visualize.py
Executable file
@@ -0,0 +1,374 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Model Evaluation Comparison Tool with Visualization
|
||||
|
||||
Extended version with plotting capabilities.
|
||||
|
||||
Usage:
|
||||
python eval_tools/compare_models_visualize.py \
|
||||
--model1 eval_results/model1/evaluation_report.json \
|
||||
--model2 eval_results/model2/evaluation_report.json \
|
||||
--output-dir comparison_results \
|
||||
--model1-name "mono3d" \
|
||||
--model2-name "yolov5s-300w"
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from eval_tools.compare_models import ModelComparator
|
||||
|
||||
try:
|
||||
import matplotlib
|
||||
matplotlib.use('Agg') # Use non-interactive backend
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
MATPLOTLIB_AVAILABLE = True
|
||||
except ImportError:
|
||||
MATPLOTLIB_AVAILABLE = False
|
||||
print("Warning: matplotlib not available, visualization will be skipped")
|
||||
|
||||
|
||||
class VisualizationComparator(ModelComparator):
|
||||
"""Extended comparator with visualization capabilities."""
|
||||
|
||||
def plot_2d_metrics_comparison(self, output_dir):
|
||||
"""Plot 2D metrics comparison."""
|
||||
if not MATPLOTLIB_AVAILABLE:
|
||||
print("Skipping 2D metrics plot (matplotlib not available)")
|
||||
return
|
||||
|
||||
print("\nGenerating 2D metrics comparison plots...")
|
||||
|
||||
comparison = self.comparison_results.get('2d_metrics', {})
|
||||
if not comparison:
|
||||
return
|
||||
|
||||
# Overall metrics bar chart
|
||||
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
|
||||
fig.suptitle('2D Detection Metrics - Overall Comparison', fontsize=16, fontweight='bold')
|
||||
|
||||
overall = comparison['overall']
|
||||
metrics = ['precision', 'recall', 'map']
|
||||
metric_names = ['Precision', 'Recall', 'mAP']
|
||||
|
||||
for idx, (metric, metric_name) in enumerate(zip(metrics, metric_names)):
|
||||
if metric not in overall:
|
||||
continue
|
||||
|
||||
values = overall[metric]
|
||||
model_names = [self.model1_name, self.model2_name]
|
||||
model_values = [values[self.model1_name], values[self.model2_name]]
|
||||
|
||||
bars = axes[idx].bar(model_names, model_values, color=['#3498db', '#e74c3c'])
|
||||
axes[idx].set_ylabel(metric_name)
|
||||
axes[idx].set_title(f'{metric_name}')
|
||||
axes[idx].set_ylim(0, 1.0)
|
||||
axes[idx].grid(axis='y', alpha=0.3)
|
||||
|
||||
# Add value labels on bars
|
||||
for bar in bars:
|
||||
height = bar.get_height()
|
||||
axes[idx].text(bar.get_x() + bar.get_width()/2., height,
|
||||
f'{height:.3f}',
|
||||
ha='center', va='bottom', fontsize=10)
|
||||
|
||||
plt.tight_layout()
|
||||
output_file = os.path.join(output_dir, 'comparison_2d_overall.png')
|
||||
plt.savefig(output_file, dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
print(f" ✓ Saved: {output_file}")
|
||||
|
||||
# Per-class AP comparison
|
||||
per_class = comparison.get('per_class', {})
|
||||
if per_class:
|
||||
class_names = sorted(per_class.keys())
|
||||
m1_aps = [per_class[c]['ap'][self.model1_name] for c in class_names]
|
||||
m2_aps = [per_class[c]['ap'][self.model2_name] for c in class_names]
|
||||
|
||||
fig, ax = plt.subplots(figsize=(12, 6))
|
||||
x = np.arange(len(class_names))
|
||||
width = 0.35
|
||||
|
||||
bars1 = ax.bar(x - width/2, m1_aps, width, label=self.model1_name, color='#3498db')
|
||||
bars2 = ax.bar(x + width/2, m2_aps, width, label=self.model2_name, color='#e74c3c')
|
||||
|
||||
ax.set_xlabel('Class', fontsize=12)
|
||||
ax.set_ylabel('Average Precision (AP)', fontsize=12)
|
||||
ax.set_title('Per-Class AP Comparison', fontsize=14, fontweight='bold')
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels(class_names, rotation=45, ha='right')
|
||||
ax.legend()
|
||||
ax.grid(axis='y', alpha=0.3)
|
||||
|
||||
plt.tight_layout()
|
||||
output_file = os.path.join(output_dir, 'comparison_2d_per_class.png')
|
||||
plt.savefig(output_file, dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
print(f" ✓ Saved: {output_file}")
|
||||
|
||||
def plot_3d_metrics_comparison(self, output_dir):
|
||||
"""Plot 3D metrics comparison."""
|
||||
if not MATPLOTLIB_AVAILABLE:
|
||||
print("Skipping 3D metrics plot (matplotlib not available)")
|
||||
return
|
||||
|
||||
print("\nGenerating 3D metrics comparison plots...")
|
||||
|
||||
comparison = self.comparison_results.get('3d_metrics', {})
|
||||
if not comparison:
|
||||
return
|
||||
|
||||
# Sort distance ranges by starting distance value
|
||||
def get_range_start(range_key):
|
||||
if range_key == 'overall':
|
||||
return -1 # Put 'overall' at the beginning
|
||||
try:
|
||||
# Extract starting distance from format like "0-20m" or "100-999m"
|
||||
return int(range_key.split('-')[0])
|
||||
except (ValueError, IndexError):
|
||||
return float('inf')
|
||||
|
||||
# For each class with distance ranges
|
||||
for class_name, ranges in comparison.items():
|
||||
if not ranges:
|
||||
continue
|
||||
|
||||
# Check if we have distance ranges, sorted by distance
|
||||
range_keys = sorted([k for k in ranges.keys() if k != 'overall'], key=get_range_start)
|
||||
if not range_keys:
|
||||
range_keys = ['overall']
|
||||
|
||||
# Create subplots for lateral, longitudinal, and heading errors
|
||||
fig, axes = plt.subplots(1, 3, figsize=(18, 5))
|
||||
fig.suptitle(f'3D Detection Metrics - {class_name.upper()}', fontsize=16, fontweight='bold')
|
||||
|
||||
error_types = ['lateral_error', 'longitudinal_error', 'heading_error']
|
||||
error_names = ['Lateral Error (m)', 'Longitudinal Error (m)', 'Heading Error (rad)']
|
||||
|
||||
for idx, (error_type, error_name) in enumerate(zip(error_types, error_names)):
|
||||
m1_values = []
|
||||
m2_values = []
|
||||
m1_stds = []
|
||||
m2_stds = []
|
||||
labels = []
|
||||
|
||||
for range_key in range_keys:
|
||||
if range_key not in ranges:
|
||||
continue
|
||||
|
||||
metrics = ranges[range_key]
|
||||
if error_type not in metrics:
|
||||
continue
|
||||
|
||||
data = metrics[error_type]
|
||||
m1_values.append(data[self.model1_name]['mean'])
|
||||
m2_values.append(data[self.model2_name]['mean'])
|
||||
m1_stds.append(data[self.model1_name]['std'])
|
||||
m2_stds.append(data[self.model2_name]['std'])
|
||||
labels.append(range_key)
|
||||
|
||||
if not m1_values:
|
||||
continue
|
||||
|
||||
x = np.arange(len(labels))
|
||||
width = 0.35
|
||||
|
||||
bars1 = axes[idx].bar(x - width/2, m1_values, width,
|
||||
yerr=m1_stds, label=self.model1_name,
|
||||
color='#3498db', alpha=0.8, capsize=5)
|
||||
bars2 = axes[idx].bar(x + width/2, m2_values, width,
|
||||
yerr=m2_stds, label=self.model2_name,
|
||||
color='#e74c3c', alpha=0.8, capsize=5)
|
||||
|
||||
axes[idx].set_xlabel('Distance Range', fontsize=10)
|
||||
axes[idx].set_ylabel(error_name, fontsize=10)
|
||||
axes[idx].set_title(error_name.split('(')[0], fontsize=12)
|
||||
axes[idx].set_xticks(x)
|
||||
axes[idx].set_xticklabels(labels, rotation=45, ha='right')
|
||||
axes[idx].legend()
|
||||
axes[idx].grid(axis='y', alpha=0.3)
|
||||
|
||||
plt.tight_layout()
|
||||
output_file = os.path.join(output_dir, f'comparison_3d_{class_name}.png')
|
||||
plt.savefig(output_file, dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
print(f" ✓ Saved: {output_file}")
|
||||
|
||||
def plot_improvement_heatmap(self, output_dir):
|
||||
"""Plot improvement heatmap for 3D metrics."""
|
||||
if not MATPLOTLIB_AVAILABLE:
|
||||
print("Skipping improvement heatmap (matplotlib not available)")
|
||||
return
|
||||
|
||||
print("\nGenerating improvement heatmap...")
|
||||
|
||||
comparison = self.comparison_results.get('3d_metrics', {})
|
||||
if not comparison:
|
||||
return
|
||||
|
||||
# Collect improvement data
|
||||
data_matrix = []
|
||||
row_labels = []
|
||||
col_labels = ['Lateral', 'Longitudinal', 'Heading']
|
||||
|
||||
# Sort distance ranges by starting distance value
|
||||
def get_range_start(range_key):
|
||||
if range_key == 'overall':
|
||||
return -1 # Put 'overall' at the beginning of each class
|
||||
try:
|
||||
# Extract starting distance from format like "0-20m" or "100-999m"
|
||||
return int(range_key.split('-')[0])
|
||||
except (ValueError, IndexError):
|
||||
return float('inf')
|
||||
|
||||
for class_name, ranges in sorted(comparison.items()):
|
||||
# Sort ranges: overall first, then by distance
|
||||
sorted_range_keys = sorted(ranges.keys(), key=get_range_start)
|
||||
for range_key in sorted_range_keys:
|
||||
metrics = ranges[range_key]
|
||||
|
||||
row_data = []
|
||||
for error_type in ['lateral_error', 'longitudinal_error', 'heading_error']:
|
||||
if error_type in metrics:
|
||||
# Negative change % means improvement (lower error)
|
||||
change = -metrics[error_type]['relative_change_%']
|
||||
row_data.append(change)
|
||||
else:
|
||||
row_data.append(0)
|
||||
|
||||
if any(x != 0 for x in row_data):
|
||||
data_matrix.append(row_data)
|
||||
label = f"{class_name}\n{range_key}"
|
||||
row_labels.append(label)
|
||||
|
||||
if not data_matrix:
|
||||
return
|
||||
|
||||
data_matrix = np.array(data_matrix)
|
||||
|
||||
fig, ax = plt.subplots(figsize=(10, max(6, len(row_labels) * 0.5)))
|
||||
|
||||
# Create heatmap
|
||||
im = ax.imshow(data_matrix, cmap='RdYlGn', aspect='auto', vmin=-50, vmax=50)
|
||||
|
||||
# Set ticks
|
||||
ax.set_xticks(np.arange(len(col_labels)))
|
||||
ax.set_yticks(np.arange(len(row_labels)))
|
||||
ax.set_xticklabels(col_labels)
|
||||
ax.set_yticklabels(row_labels, fontsize=8)
|
||||
|
||||
# Add colorbar
|
||||
cbar = plt.colorbar(im, ax=ax)
|
||||
cbar.set_label(f'Improvement % ({self.model2_name} vs {self.model1_name})', rotation=270, labelpad=20)
|
||||
|
||||
# Add text annotations
|
||||
for i in range(len(row_labels)):
|
||||
for j in range(len(col_labels)):
|
||||
text = ax.text(j, i, f'{data_matrix[i, j]:.1f}%',
|
||||
ha="center", va="center", color="black", fontsize=8)
|
||||
|
||||
ax.set_title(f'3D Metrics Improvement Heatmap\n(Positive = {self.model2_name} Better)',
|
||||
fontsize=14, fontweight='bold')
|
||||
|
||||
plt.tight_layout()
|
||||
output_file = os.path.join(output_dir, 'comparison_3d_improvement_heatmap.png')
|
||||
plt.savefig(output_file, dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
print(f" ✓ Saved: {output_file}")
|
||||
|
||||
def generate_visualizations(self, output_dir):
|
||||
"""Generate all visualizations."""
|
||||
if not MATPLOTLIB_AVAILABLE:
|
||||
print("\n⚠ Matplotlib not available, skipping visualizations")
|
||||
print("Install with: pip install matplotlib")
|
||||
return
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("GENERATING VISUALIZATIONS")
|
||||
print("="*80)
|
||||
|
||||
self.plot_2d_metrics_comparison(output_dir)
|
||||
self.plot_3d_metrics_comparison(output_dir)
|
||||
self.plot_improvement_heatmap(output_dir)
|
||||
|
||||
print("\n✓ All visualizations generated")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Compare evaluation results from two models with visualization',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter
|
||||
)
|
||||
|
||||
parser.add_argument('--model1', type=str, required=True,
|
||||
help='Path to model 1 evaluation report JSON')
|
||||
parser.add_argument('--model2', type=str, required=True,
|
||||
help='Path to model 2 evaluation report JSON')
|
||||
parser.add_argument('--output-dir', type=str, default='comparison_results',
|
||||
help='Output directory for comparison results')
|
||||
parser.add_argument('--model1-name', type=str, default='Model-1',
|
||||
help='Display name for model 1')
|
||||
parser.add_argument('--model2-name', type=str, default='Model-2',
|
||||
help='Display name for model 2')
|
||||
parser.add_argument('--no-plots', action='store_true',
|
||||
help='Skip visualization generation')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Load reports
|
||||
print("="*80)
|
||||
print("MODEL COMPARISON TOOL (with Visualization)")
|
||||
print("="*80)
|
||||
print(f"\nLoading model 1: {args.model1}")
|
||||
with open(args.model1, 'r') as f:
|
||||
model1_report = json.load(f)
|
||||
|
||||
print(f"Loading model 2: {args.model2}")
|
||||
with open(args.model2, 'r') as f:
|
||||
model2_report = json.load(f)
|
||||
|
||||
# Create output directory
|
||||
os.makedirs(args.output_dir, exist_ok=True)
|
||||
|
||||
# Compare models
|
||||
comparator = VisualizationComparator(
|
||||
model1_report,
|
||||
model2_report,
|
||||
model1_name=args.model1_name,
|
||||
model2_name=args.model2_name
|
||||
)
|
||||
|
||||
results = comparator.compare_all()
|
||||
|
||||
# Generate reports
|
||||
text_output = os.path.join(args.output_dir, 'comparison_report.txt')
|
||||
json_output = os.path.join(args.output_dir, 'comparison_report.json')
|
||||
|
||||
comparator.generate_text_report(text_output)
|
||||
comparator.generate_json_report(json_output)
|
||||
|
||||
# Generate visualizations
|
||||
if not args.no_plots:
|
||||
comparator.generate_visualizations(args.output_dir)
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("COMPARISON COMPLETE")
|
||||
print("="*80)
|
||||
print(f"\nResults saved to: {args.output_dir}/")
|
||||
print(f" - Text report: comparison_report.txt")
|
||||
print(f" - JSON report: comparison_report.json")
|
||||
if not args.no_plots and MATPLOTLIB_AVAILABLE:
|
||||
print(f" - Visualization plots: comparison_*.png")
|
||||
print("")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
185
eval_tools/model_comparison/compare_models_with_common_matches.sh
Executable file
185
eval_tools/model_comparison/compare_models_with_common_matches.sh
Executable file
@@ -0,0 +1,185 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Common Match Comparison Workflow Example
|
||||
#
|
||||
# This script demonstrates the complete workflow for comparing two models
|
||||
# using only the GT objects that both models successfully matched.
|
||||
#
|
||||
# Usage:
|
||||
# bash eval_tools/compare_models_with_common_matches.sh
|
||||
|
||||
set -e # Exit on error
|
||||
|
||||
# Set PYTHONPATH to project root for module imports
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
|
||||
|
||||
echo "================================================================================"
|
||||
echo "Common Match Comparison Workflow"
|
||||
echo "================================================================================"
|
||||
|
||||
# Configuration
|
||||
MODEL1_CONFIG="eval_tools/configs/eval_config_yolov5s_cncap_768-roi1.yaml"
|
||||
MODEL2_CONFIG="eval_tools/configs/eval_config_yolov5s_cncap-roi1.yaml"
|
||||
OUTPUT_BASE="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_768_roi1-conf0.4-v2"
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
MODEL1_NAME="20260317"
|
||||
MODEL2_NAME="20260228"
|
||||
|
||||
# Heading tolerance mode: strict, relaxed, or both
|
||||
HEADING_TOLERANCE="both"
|
||||
|
||||
# Step 1: Evaluate Model 1 with detailed match saving
|
||||
echo ""
|
||||
echo "Step 1: Evaluating Model 1 (${MODEL1_NAME}) with detailed match saving..."
|
||||
echo " Heading tolerance: ${HEADING_TOLERANCE}"
|
||||
echo "--------------------------------------------------------------------------------"
|
||||
MODEL1_OUTPUT="${OUTPUT_BASE}/${MODEL1_NAME}/${TIMESTAMP}"
|
||||
python eval_tools/core/eval.py \
|
||||
--config ${MODEL1_CONFIG} \
|
||||
--output-dir ${MODEL1_OUTPUT} \
|
||||
--heading-tolerance ${HEADING_TOLERANCE} \
|
||||
--save-detailed-matches
|
||||
|
||||
# Use the output directory we specified
|
||||
MODEL1_DIR=${MODEL1_OUTPUT}
|
||||
if [ ! -d "$MODEL1_DIR" ]; then
|
||||
echo "Error: Could not find Model 1 evaluation results at ${MODEL1_DIR}"
|
||||
exit 1
|
||||
fi
|
||||
echo "✓ Model 1 results: ${MODEL1_DIR}"
|
||||
|
||||
# Generate Markdown report for Model 1
|
||||
python ${SCRIPT_DIR}/generate_eval_report.py \
|
||||
${MODEL1_DIR}/evaluation_report.json \
|
||||
--model "${MODEL1_NAME}" \
|
||||
--date $(date +%Y-%m-%d)
|
||||
echo "✓ Model 1 Markdown report: ${MODEL1_DIR}/EVALUATION_REPORT.md"
|
||||
echo ""
|
||||
echo "Step 2: Evaluating Model 2 (${MODEL2_NAME}) with detailed match saving..."
|
||||
echo " Heading tolerance: ${HEADING_TOLERANCE}"
|
||||
echo "--------------------------------------------------------------------------------"
|
||||
MODEL2_OUTPUT="${OUTPUT_BASE}/${MODEL2_NAME}/${TIMESTAMP}"
|
||||
python eval_tools/core/eval.py \
|
||||
--config ${MODEL2_CONFIG} \
|
||||
--output-dir ${MODEL2_OUTPUT} \
|
||||
--heading-tolerance ${HEADING_TOLERANCE} \
|
||||
--save-detailed-matches
|
||||
|
||||
# Use the output directory we specified
|
||||
MODEL2_DIR=${MODEL2_OUTPUT}
|
||||
if [ ! -d "$MODEL2_DIR" ]; then
|
||||
echo "Error: Could not find Model 2 evaluation results at ${MODEL2_DIR}"
|
||||
exit 1
|
||||
fi
|
||||
echo "✓ Model 2 results: ${MODEL2_DIR}"
|
||||
|
||||
# Generate Markdown report for Model 2
|
||||
python ${SCRIPT_DIR}/generate_eval_report.py \
|
||||
${MODEL2_DIR}/evaluation_report.json \
|
||||
--model "${MODEL2_NAME}" \
|
||||
--date $(date +%Y-%m-%d)
|
||||
echo "✓ Model 2 Markdown report: ${MODEL2_DIR}/EVALUATION_REPORT.md"
|
||||
|
||||
# Step 3: Find common matches
|
||||
echo ""
|
||||
echo "Step 3: Finding common matches between the two models..."
|
||||
echo "--------------------------------------------------------------------------------"
|
||||
COMMON_MATCHES_DIR="${OUTPUT_BASE}/common_matches_${TIMESTAMP}"
|
||||
mkdir -p ${COMMON_MATCHES_DIR}
|
||||
|
||||
python eval_tools/model_comparison/find_common_matches.py \
|
||||
--model1-matches ${MODEL1_DIR}/detailed_3d_matches.json \
|
||||
--model2-matches ${MODEL2_DIR}/detailed_3d_matches.json \
|
||||
--output ${COMMON_MATCHES_DIR}/common_matches.json \
|
||||
--model1-name "${MODEL1_NAME}" \
|
||||
--model2-name "${MODEL2_NAME}"
|
||||
|
||||
echo "✓ Common matches saved to: ${COMMON_MATCHES_DIR}/common_matches.json"
|
||||
|
||||
# Step 4: Compare models using common matches
|
||||
echo ""
|
||||
echo "Step 4: Comparing models using common matches only..."
|
||||
echo "--------------------------------------------------------------------------------"
|
||||
COMPARISON_DIR="${OUTPUT_BASE}/comparison_common_matches_${TIMESTAMP}"
|
||||
|
||||
python eval_tools/model_comparison/compare_models.py \
|
||||
--model1 ${MODEL1_DIR}/evaluation_report.json \
|
||||
--model2 ${MODEL2_DIR}/evaluation_report.json \
|
||||
--model1-name "${MODEL1_NAME}" \
|
||||
--model2-name "${MODEL2_NAME}" \
|
||||
--common-matches ${COMMON_MATCHES_DIR}/common_matches.json \
|
||||
--output-dir ${COMPARISON_DIR}
|
||||
|
||||
echo "✓ Comparison results saved to: ${COMPARISON_DIR}"
|
||||
|
||||
# Generate Markdown report for common-match comparison
|
||||
python ${SCRIPT_DIR}/generate_comparison_report.py \
|
||||
${COMPARISON_DIR}/comparison_report.json \
|
||||
--date $(date +%Y-%m-%d)
|
||||
echo "✓ Markdown report: ${COMPARISON_DIR}/COMPARISON_REPORT.md"
|
||||
|
||||
# Step 5: Also run traditional comparison (without common match filtering)
|
||||
echo ""
|
||||
echo "Step 5: Running traditional comparison (all matches) for reference..."
|
||||
echo "--------------------------------------------------------------------------------"
|
||||
COMPARISON_TRADITIONAL_DIR="${OUTPUT_BASE}/comparison_all_matches_${TIMESTAMP}"
|
||||
|
||||
python eval_tools/model_comparison/compare_models.py \
|
||||
--model1 ${MODEL1_DIR}/evaluation_report.json \
|
||||
--model2 ${MODEL2_DIR}/evaluation_report.json \
|
||||
--model1-name "${MODEL1_NAME}" \
|
||||
--model2-name "${MODEL2_NAME}" \
|
||||
--output-dir ${COMPARISON_TRADITIONAL_DIR}
|
||||
|
||||
echo "✓ Traditional comparison saved to: ${COMPARISON_TRADITIONAL_DIR}"
|
||||
|
||||
# Generate Markdown report for traditional comparison
|
||||
python ${SCRIPT_DIR}/generate_comparison_report.py \
|
||||
${COMPARISON_TRADITIONAL_DIR}/comparison_report.json \
|
||||
--date $(date +%Y-%m-%d)
|
||||
echo "✓ Markdown report: ${COMPARISON_TRADITIONAL_DIR}/COMPARISON_REPORT.md"
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
echo "================================================================================"
|
||||
echo "WORKFLOW COMPLETE!"
|
||||
echo "================================================================================"
|
||||
echo ""
|
||||
echo "Results Summary:"
|
||||
echo " Model 1 (${MODEL1_NAME}):"
|
||||
echo " - Evaluation: ${MODEL1_DIR}"
|
||||
echo " - Detailed matches: ${MODEL1_DIR}/detailed_3d_matches.json"
|
||||
echo " - Markdown report: ${MODEL1_DIR}/EVALUATION_REPORT.md"
|
||||
echo ""
|
||||
echo " Model 2 (${MODEL2_NAME}):"
|
||||
echo " - Evaluation: ${MODEL2_DIR}"
|
||||
echo " - Detailed matches: ${MODEL2_DIR}/detailed_3d_matches.json"
|
||||
echo " - Markdown report: ${MODEL2_DIR}/EVALUATION_REPORT.md"
|
||||
echo ""
|
||||
echo " Common Matches Analysis:"
|
||||
echo " - Common matches data: ${COMMON_MATCHES_DIR}/common_matches.json"
|
||||
echo ""
|
||||
echo " Comparison Results:"
|
||||
echo " - Common matches only: ${COMPARISON_DIR}/"
|
||||
echo " - All matches (traditional): ${COMPARISON_TRADITIONAL_DIR}/"
|
||||
echo ""
|
||||
echo "Key Files to Review:"
|
||||
echo " 1. ${COMPARISON_DIR}/comparison_report.txt"
|
||||
echo " (3D comparison based on common matches - fair comparison)"
|
||||
echo " 1b. ${COMPARISON_DIR}/COMPARISON_REPORT.md"
|
||||
echo " (Markdown report - common matches)"
|
||||
echo ""
|
||||
echo " 2. ${COMPARISON_TRADITIONAL_DIR}/comparison_report.txt"
|
||||
echo " (Traditional comparison with all matches - for reference)"
|
||||
echo " 2b. ${COMPARISON_TRADITIONAL_DIR}/COMPARISON_REPORT.md"
|
||||
echo " (Markdown report - all matches)"
|
||||
echo ""
|
||||
echo " 3. ${COMMON_MATCHES_DIR}/common_matches.json"
|
||||
echo " (Detailed statistics about match differences)"
|
||||
echo ""
|
||||
echo "To view the common-match comparison report:"
|
||||
echo " cat ${COMPARISON_DIR}/comparison_report.txt"
|
||||
echo ""
|
||||
285
eval_tools/model_comparison/compare_per_case_2d.py
Executable file
285
eval_tools/model_comparison/compare_per_case_2d.py
Executable file
@@ -0,0 +1,285 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Per-Case 2D Metrics Comparison Tool
|
||||
|
||||
This script compares per_case_2d metrics from two model evaluation reports
|
||||
and identifies cases with significant metric differences.
|
||||
|
||||
Usage:
|
||||
python eval_tools/model_comparison/compare_per_case_2d.py \
|
||||
--model1 evaluation_results/.../evaluation_report.json \
|
||||
--model2 evaluation_results/.../evaluation_report.json \
|
||||
--threshold 0.1 \
|
||||
--output comparison_per_case_2d.json
|
||||
|
||||
Example:
|
||||
python eval_tools/model_comparison/compare_per_case_2d.py \
|
||||
--model1 evaluation_results/eval_results_common_match_comparison_CNCAP_roi0/mono3d/20260211_113153/evaluation_report.json \
|
||||
--model2 evaluation_results/eval_results_common_match_comparison_CNCAP_roi0/yolov5s-300w-newdata/20260211_113153/evaluation_report.json \
|
||||
--threshold 0.1 \
|
||||
--output per_case_comparison.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
import sys
|
||||
import numpy as np
|
||||
|
||||
# Allow importing class_config from the eval_tools root
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
from class_config import CLASS_NAMES
|
||||
|
||||
|
||||
class PerCaseComparator:
|
||||
"""Compare per_case_2d metrics between two models."""
|
||||
|
||||
def __init__(self, model1_report, model2_report, model1_name="Model-1", model2_name="Model-2"):
|
||||
"""
|
||||
Initialize comparator.
|
||||
|
||||
Args:
|
||||
model1_report: dict, evaluation report for model 1
|
||||
model2_report: dict, evaluation report for model 2
|
||||
model1_name: str, display name for model 1
|
||||
model2_name: str, display name for model 2
|
||||
"""
|
||||
self.model1_report = model1_report
|
||||
self.model2_report = model2_report
|
||||
self.model1_name = model1_name
|
||||
self.model2_name = model2_name
|
||||
|
||||
def compare_per_case_metrics(self, threshold=0.1, metric_name='ap'):
|
||||
"""
|
||||
Compare per_case_2d metrics and identify cases with significant differences.
|
||||
|
||||
Args:
|
||||
threshold: float, threshold for significant difference (default 0.1 = 10%)
|
||||
metric_name: str, metric to compare ('ap', 'precision', 'recall')
|
||||
|
||||
Returns:
|
||||
dict with comparison results
|
||||
"""
|
||||
print(f"\n{'='*80}")
|
||||
print(f"Comparing Per-Case 2D Metrics (threshold={threshold*100}%)")
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
# Get per_case_2d data
|
||||
m1_cases = self.model1_report.get('per_case_2d', {})
|
||||
m2_cases = self.model2_report.get('per_case_2d', {})
|
||||
|
||||
# Find common cases
|
||||
common_cases = set(m1_cases.keys()) & set(m2_cases.keys())
|
||||
print(f"Total cases in {self.model1_name}: {len(m1_cases)}")
|
||||
print(f"Total cases in {self.model2_name}: {len(m2_cases)}")
|
||||
print(f"Common cases: {len(common_cases)}\n")
|
||||
|
||||
# Compare each case
|
||||
case_comparisons = {}
|
||||
significant_diffs = []
|
||||
|
||||
for case_name in sorted(common_cases):
|
||||
m1_case = m1_cases[case_name]
|
||||
m2_case = m2_cases[case_name]
|
||||
|
||||
case_comp = {
|
||||
'case_name': case_name,
|
||||
'per_class': {},
|
||||
'max_diff': 0.0,
|
||||
'max_diff_class': None,
|
||||
'max_diff_metric': None
|
||||
}
|
||||
|
||||
# Compare per-class metrics
|
||||
m1_classes = m1_case.get('per_class', {})
|
||||
m2_classes = m2_case.get('per_class', {})
|
||||
|
||||
for class_name in m1_classes.keys():
|
||||
if class_name not in m2_classes:
|
||||
continue
|
||||
|
||||
m1_class = m1_classes[class_name]
|
||||
m2_class = m2_classes[class_name]
|
||||
|
||||
class_comp = {}
|
||||
for metric in ['precision', 'recall', 'ap']:
|
||||
m1_val = m1_class.get(metric, 0.0)
|
||||
m2_val = m2_class.get(metric, 0.0)
|
||||
diff = m2_val - m1_val
|
||||
|
||||
class_comp[metric] = {
|
||||
self.model1_name: m1_val,
|
||||
self.model2_name: m2_val,
|
||||
'diff': diff,
|
||||
'abs_diff': abs(diff)
|
||||
}
|
||||
|
||||
# Track maximum difference
|
||||
if abs(diff) > case_comp['max_diff']:
|
||||
case_comp['max_diff'] = abs(diff)
|
||||
case_comp['max_diff_class'] = class_name
|
||||
case_comp['max_diff_metric'] = metric
|
||||
|
||||
# Add count metrics for context
|
||||
class_comp['counts'] = {
|
||||
'num_gt': m1_class.get('num_gt', 0),
|
||||
'num_det_m1': m1_class.get('num_det', 0),
|
||||
'num_det_m2': m2_class.get('num_det', 0),
|
||||
}
|
||||
|
||||
case_comp['per_class'][class_name] = class_comp
|
||||
|
||||
case_comparisons[case_name] = case_comp
|
||||
|
||||
# Check if this case has significant differences
|
||||
if case_comp['max_diff'] >= threshold:
|
||||
significant_diffs.append({
|
||||
'case_name': case_name,
|
||||
'max_diff': case_comp['max_diff'],
|
||||
'class': case_comp['max_diff_class'],
|
||||
'metric': case_comp['max_diff_metric'],
|
||||
'details': case_comp['per_class'][case_comp['max_diff_class']][case_comp['max_diff_metric']]
|
||||
})
|
||||
|
||||
# Sort by maximum difference
|
||||
significant_diffs.sort(key=lambda x: x['max_diff'], reverse=True)
|
||||
|
||||
results = {
|
||||
'summary': {
|
||||
'total_common_cases': len(common_cases),
|
||||
'cases_with_significant_diff': len(significant_diffs),
|
||||
'threshold': threshold,
|
||||
'model1_name': self.model1_name,
|
||||
'model2_name': self.model2_name
|
||||
},
|
||||
'significant_differences': significant_diffs,
|
||||
'all_case_comparisons': case_comparisons
|
||||
}
|
||||
|
||||
return results
|
||||
|
||||
def print_significant_differences(self, results, top_n=20):
|
||||
"""Print top N cases with significant differences."""
|
||||
sig_diffs = results['significant_differences']
|
||||
|
||||
print(f"\n{'='*80}")
|
||||
print(f"Top {min(top_n, len(sig_diffs))} Cases with Significant Differences")
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
for i, diff in enumerate(sig_diffs[:top_n], 1):
|
||||
details = diff['details']
|
||||
print(f"{i}. Case: {diff['case_name']}")
|
||||
print(f" Class: {diff['class']}, Metric: {diff['metric']}")
|
||||
print(f" {self.model1_name}: {details[self.model1_name]:.4f}")
|
||||
print(f" {self.model2_name}: {details[self.model2_name]:.4f}")
|
||||
print(f" Difference: {details['diff']:+.4f} (abs: {diff['max_diff']:.4f})")
|
||||
print()
|
||||
|
||||
def generate_summary_stats(self, results):
|
||||
"""Generate summary statistics."""
|
||||
all_comps = results['all_case_comparisons']
|
||||
|
||||
# Collect all differences by class and metric
|
||||
diffs_by_class_metric = defaultdict(list)
|
||||
|
||||
for case_name, case_comp in all_comps.items():
|
||||
for class_name, class_comp in case_comp['per_class'].items():
|
||||
for metric in ['precision', 'recall', 'ap']:
|
||||
diff = class_comp[metric]['diff']
|
||||
diffs_by_class_metric[(class_name, metric)].append(diff)
|
||||
|
||||
# Calculate statistics
|
||||
stats = {}
|
||||
for (class_name, metric), diffs in diffs_by_class_metric.items():
|
||||
diffs_array = np.array(diffs)
|
||||
stats[f"{class_name}_{metric}"] = {
|
||||
'mean_diff': float(np.mean(diffs_array)),
|
||||
'std_diff': float(np.std(diffs_array)),
|
||||
'median_diff': float(np.median(diffs_array)),
|
||||
'min_diff': float(np.min(diffs_array)),
|
||||
'max_diff': float(np.max(diffs_array)),
|
||||
'num_cases': len(diffs)
|
||||
}
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Compare per_case_2d metrics between two model evaluation reports'
|
||||
)
|
||||
parser.add_argument('--model1', type=str, required=True,
|
||||
help='Path to model 1 evaluation_report.json')
|
||||
parser.add_argument('--model2', type=str, required=True,
|
||||
help='Path to model 2 evaluation_report.json')
|
||||
parser.add_argument('--model1-name', type=str, default='Model-1',
|
||||
help='Display name for model 1')
|
||||
parser.add_argument('--model2-name', type=str, default='Model-2',
|
||||
help='Display name for model 2')
|
||||
parser.add_argument('--threshold', type=float, default=0.1,
|
||||
help='Threshold for significant difference (default: 0.1 = 10%%)')
|
||||
parser.add_argument('--output', type=str, default='per_case_comparison.json',
|
||||
help='Output JSON file path')
|
||||
parser.add_argument('--top-n', type=int, default=20,
|
||||
help='Number of top different cases to display (default: 20)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Load evaluation reports
|
||||
print(f"Loading {args.model1}...")
|
||||
with open(args.model1, 'r') as f:
|
||||
model1_report = json.load(f)
|
||||
|
||||
print(f"Loading {args.model2}...")
|
||||
with open(args.model2, 'r') as f:
|
||||
model2_report = json.load(f)
|
||||
|
||||
# Create comparator
|
||||
comparator = PerCaseComparator(
|
||||
model1_report, model2_report,
|
||||
model1_name=args.model1_name,
|
||||
model2_name=args.model2_name
|
||||
)
|
||||
|
||||
# Compare metrics
|
||||
results = comparator.compare_per_case_metrics(threshold=args.threshold)
|
||||
|
||||
# Print significant differences
|
||||
comparator.print_significant_differences(results, top_n=args.top_n)
|
||||
|
||||
# Generate summary statistics
|
||||
print(f"\n{'='*80}")
|
||||
print("Summary Statistics")
|
||||
print(f"{'='*80}\n")
|
||||
stats = comparator.generate_summary_stats(results)
|
||||
|
||||
# Print stats for main classes (all 3D classes, skipping vehicle sub-buckets)
|
||||
for class_name in [n for n in CLASS_NAMES.values() if n in stats or f"{n}_ap" in stats]:
|
||||
print(f"\n{class_name.upper()}:")
|
||||
for metric in ['ap', 'precision', 'recall']:
|
||||
key = f"{class_name}_{metric}"
|
||||
if key in stats:
|
||||
s = stats[key]
|
||||
print(f" {metric:10s}: mean={s['mean_diff']:+.4f}, "
|
||||
f"std={s['std_diff']:.4f}, median={s['median_diff']:+.4f}, "
|
||||
f"range=[{s['min_diff']:+.4f}, {s['max_diff']:+.4f}]")
|
||||
|
||||
# Save results
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Add summary stats to results
|
||||
results['summary_statistics'] = stats
|
||||
|
||||
with open(output_path, 'w') as f:
|
||||
json.dump(results, f, indent=2)
|
||||
|
||||
print(f"\n{'='*80}")
|
||||
print(f"Results saved to: {output_path}")
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
26
eval_tools/model_comparison/compare_per_case_2d.sh
Executable file
26
eval_tools/model_comparison/compare_per_case_2d.sh
Executable file
@@ -0,0 +1,26 @@
|
||||
#!/bin/bash
|
||||
# Compare per_case_2d metrics between mono3d and yolov5s-300w-newdata models
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
|
||||
MODEL1_PATH="$PROJECT_ROOT/evaluation_results/eval_results_common_match_comparison_CNCAP_roi1/mono3d/20260211_120939/evaluation_report.json"
|
||||
MODEL2_PATH="$PROJECT_ROOT/evaluation_results/eval_results_common_match_comparison_CNCAP_roi1/yolov5s-300w-newdata/20260211_120939/evaluation_report.json"
|
||||
OUTPUT_PATH="$PROJECT_ROOT/evaluation_results/eval_results_common_match_comparison_CNCAP_roi1/per_case_2d_comparison.json"
|
||||
|
||||
echo "Comparing per_case_2d metrics..."
|
||||
echo "Model 1: mono3d"
|
||||
echo "Model 2: yolov5s-300w-newdata"
|
||||
echo ""
|
||||
|
||||
python "$SCRIPT_DIR/compare_per_case_2d.py" \
|
||||
--model1 "$MODEL1_PATH" \
|
||||
--model2 "$MODEL2_PATH" \
|
||||
--model1-name "mono3d" \
|
||||
--model2-name "yolov5s-300w-newdata" \
|
||||
--threshold 0.1 \
|
||||
--output "$OUTPUT_PATH" \
|
||||
--top-n 30
|
||||
|
||||
echo ""
|
||||
echo "Done! Results saved to: $OUTPUT_PATH"
|
||||
439
eval_tools/model_comparison/find_common_matches.py
Executable file
439
eval_tools/model_comparison/find_common_matches.py
Executable file
@@ -0,0 +1,439 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Find Common Matches Between Two Models
|
||||
|
||||
This tool finds the common GT objects that were successfully matched by both models,
|
||||
enabling fair comparison of 3D prediction quality on the same set of targets.
|
||||
|
||||
Usage:
|
||||
python eval_tools/find_common_matches.py \
|
||||
--model1-matches eval_results/model1/detailed_3d_matches.json \
|
||||
--model2-matches eval_results/model2/detailed_3d_matches.json \
|
||||
--output common_matches.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
import numpy as np
|
||||
|
||||
|
||||
def find_common_matches(model1_matches, model2_matches):
|
||||
"""
|
||||
Find common GT objects matched by both models.
|
||||
|
||||
Args:
|
||||
model1_matches: dict, detailed matches from model 1
|
||||
model2_matches: dict, detailed matches from model 2
|
||||
|
||||
Returns:
|
||||
tuple: (common_matches, stats)
|
||||
common_matches: dict with structure {case: {frame: {class: [match_info]}}}
|
||||
stats: dict with match statistics
|
||||
"""
|
||||
common_matches = {}
|
||||
stats = {
|
||||
'model1_total': 0,
|
||||
'model2_total': 0,
|
||||
'common': 0,
|
||||
'model1_unique': 0,
|
||||
'model2_unique': 0,
|
||||
'per_class': {}
|
||||
}
|
||||
|
||||
# Iterate through all cases in model 1
|
||||
for case_name in model1_matches:
|
||||
if case_name not in model2_matches:
|
||||
# Case not in model 2, skip
|
||||
continue
|
||||
|
||||
common_matches[case_name] = {}
|
||||
|
||||
# Iterate through frames
|
||||
for frame_name in model1_matches[case_name]:
|
||||
if frame_name not in model2_matches[case_name]:
|
||||
# Frame not in model 2, skip
|
||||
continue
|
||||
|
||||
common_matches[case_name][frame_name] = {}
|
||||
|
||||
# Iterate through classes
|
||||
for class_name in model1_matches[case_name][frame_name]:
|
||||
if class_name not in model2_matches[case_name][frame_name]:
|
||||
# Class not in model 2, skip
|
||||
continue
|
||||
|
||||
# Get match lists for this class
|
||||
m1_list = model1_matches[case_name][frame_name][class_name]
|
||||
m2_list = model2_matches[case_name][frame_name][class_name]
|
||||
|
||||
# Build GT ID to index mappings
|
||||
m1_gt_ids = {m['gt_id']: i for i, m in enumerate(m1_list)}
|
||||
m2_gt_ids = {m['gt_id']: i for i, m in enumerate(m2_list)}
|
||||
|
||||
# Find common GT IDs
|
||||
common_gt_ids = set(m1_gt_ids.keys()) & set(m2_gt_ids.keys())
|
||||
|
||||
# Update statistics
|
||||
if class_name not in stats['per_class']:
|
||||
stats['per_class'][class_name] = {
|
||||
'model1_total': 0,
|
||||
'model2_total': 0,
|
||||
'common': 0,
|
||||
'model1_unique': 0,
|
||||
'model2_unique': 0
|
||||
}
|
||||
|
||||
stats['model1_total'] += len(m1_list)
|
||||
stats['model2_total'] += len(m2_list)
|
||||
stats['common'] += len(common_gt_ids)
|
||||
stats['model1_unique'] += len(m1_gt_ids) - len(common_gt_ids)
|
||||
stats['model2_unique'] += len(m2_gt_ids) - len(common_gt_ids)
|
||||
|
||||
stats['per_class'][class_name]['model1_total'] += len(m1_list)
|
||||
stats['per_class'][class_name]['model2_total'] += len(m2_list)
|
||||
stats['per_class'][class_name]['common'] += len(common_gt_ids)
|
||||
stats['per_class'][class_name]['model1_unique'] += len(m1_gt_ids) - len(common_gt_ids)
|
||||
stats['per_class'][class_name]['model2_unique'] += len(m2_gt_ids) - len(common_gt_ids)
|
||||
|
||||
# Store common match information
|
||||
common_list = []
|
||||
for gt_id in common_gt_ids:
|
||||
common_list.append({
|
||||
'gt_id': gt_id,
|
||||
'model1_idx': m1_gt_ids[gt_id],
|
||||
'model2_idx': m2_gt_ids[gt_id]
|
||||
})
|
||||
|
||||
if common_list:
|
||||
common_matches[case_name][frame_name][class_name] = common_list
|
||||
|
||||
# Calculate percentages
|
||||
if stats['model1_total'] > 0:
|
||||
stats['common_percentage_of_model1'] = (stats['common'] / stats['model1_total']) * 100
|
||||
else:
|
||||
stats['common_percentage_of_model1'] = 0
|
||||
|
||||
if stats['model2_total'] > 0:
|
||||
stats['common_percentage_of_model2'] = (stats['common'] / stats['model2_total']) * 100
|
||||
else:
|
||||
stats['common_percentage_of_model2'] = 0
|
||||
|
||||
for class_name in stats['per_class']:
|
||||
class_stats = stats['per_class'][class_name]
|
||||
if class_stats['model1_total'] > 0:
|
||||
class_stats['common_percentage_of_model1'] = (class_stats['common'] / class_stats['model1_total']) * 100
|
||||
else:
|
||||
class_stats['common_percentage_of_model1'] = 0
|
||||
|
||||
if class_stats['model2_total'] > 0:
|
||||
class_stats['common_percentage_of_model2'] = (class_stats['common'] / class_stats['model2_total']) * 100
|
||||
else:
|
||||
class_stats['common_percentage_of_model2'] = 0
|
||||
|
||||
return common_matches, stats
|
||||
|
||||
|
||||
# Default distance ranges matching eval metrics_3d config
|
||||
DEFAULT_LONG_RANGES = [
|
||||
(0, 10), (10, 20), (20, 30), (30, 40), (40, 50),
|
||||
(50, 60), (60, 70), (70, 80), (80, 90), (90, 100), (100, 999)
|
||||
]
|
||||
DEFAULT_LAT_RANGES = [
|
||||
(-50, -40), (-40, -30), (-30, -20), (-20, -10), (-10, 0),
|
||||
(0, 10), (10, 20), (20, 30), (30, 40), (40, 50)
|
||||
]
|
||||
|
||||
|
||||
def _range_key_long(lo, hi):
|
||||
return f'long_{lo}-{hi}m'
|
||||
|
||||
|
||||
def _range_key_lat(lo, hi):
|
||||
return f'lat_{lo}-{hi}m'
|
||||
|
||||
|
||||
def _make_stats(data_dict):
|
||||
"""Compute mean/std/median/min/max for each list in data_dict."""
|
||||
result = {}
|
||||
for key, values in data_dict.items():
|
||||
if key in ('samples',):
|
||||
result[key] = values
|
||||
elif isinstance(values, list) and len(values) > 0:
|
||||
arr = np.array(values)
|
||||
result[key] = {
|
||||
'mean': float(np.mean(arr)),
|
||||
'std': float(np.std(arr)),
|
||||
'median': float(np.median(arr)),
|
||||
'min': float(np.min(arr)),
|
||||
'max': float(np.max(arr)),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def _empty_bucket():
|
||||
return {
|
||||
'lateral': [], 'longitudinal': [], 'longitudinal_relative': [],
|
||||
'heading': [], 'heading_relaxed': [], 'is_reversal': [], 'samples': 0
|
||||
}
|
||||
|
||||
|
||||
def _finalize_class_stats(data):
|
||||
"""Convert a bucket dict to stats dict, adding optional fields."""
|
||||
entry = {
|
||||
'num_samples': data['samples'],
|
||||
'lateral_error': _make_stats({'lateral': data['lateral']})['lateral'],
|
||||
'longitudinal_error': _make_stats({'longitudinal': data['longitudinal']})['longitudinal'],
|
||||
'heading_error': _make_stats({'heading': data['heading']})['heading'],
|
||||
}
|
||||
if data['longitudinal_relative']:
|
||||
entry['longitudinal_relative_error'] = _make_stats(
|
||||
{'v': data['longitudinal_relative']})['v']
|
||||
if data['heading_relaxed']:
|
||||
entry['heading_error_relaxed'] = _make_stats(
|
||||
{'v': data['heading_relaxed']})['v']
|
||||
if data['is_reversal']:
|
||||
count = int(sum(data['is_reversal']))
|
||||
entry['reversal_count'] = count
|
||||
entry['reversal_percentage'] = float(count / data['samples'] * 100) if data['samples'] > 0 else 0.0
|
||||
return entry
|
||||
|
||||
|
||||
def recompute_3d_stats_from_common_matches(matches_data, common_matches, model_name,
|
||||
long_ranges=None, lat_ranges=None):
|
||||
"""
|
||||
Recompute 3D statistics based on common matches only.
|
||||
|
||||
Returns a dict with structure:
|
||||
{
|
||||
class_name: {
|
||||
'overall': { ... },
|
||||
'long_0-10m': { ... },
|
||||
...
|
||||
'lat_-10-0m': { ... },
|
||||
...
|
||||
}
|
||||
}
|
||||
"""
|
||||
if long_ranges is None:
|
||||
long_ranges = DEFAULT_LONG_RANGES
|
||||
if lat_ranges is None:
|
||||
lat_ranges = DEFAULT_LAT_RANGES
|
||||
|
||||
# Bucket structure: class -> range_key -> _empty_bucket()
|
||||
overall = {} # class -> _empty_bucket()
|
||||
by_long = {} # class -> range_key -> _empty_bucket()
|
||||
by_lat = {} # class -> range_key -> _empty_bucket()
|
||||
|
||||
for case_name, frames in common_matches.items():
|
||||
for frame_name, classes in frames.items():
|
||||
for class_name, common_list in classes.items():
|
||||
if class_name not in overall:
|
||||
overall[class_name] = _empty_bucket()
|
||||
by_long[class_name] = {
|
||||
_range_key_long(lo, hi): _empty_bucket()
|
||||
for lo, hi in long_ranges
|
||||
}
|
||||
by_lat[class_name] = {
|
||||
_range_key_lat(lo, hi): _empty_bucket()
|
||||
for lo, hi in lat_ranges
|
||||
}
|
||||
|
||||
for match_info in common_list:
|
||||
idx = match_info[f'{model_name}_idx']
|
||||
match = matches_data[case_name][frame_name][class_name][idx]
|
||||
errs = match['errors']
|
||||
dist = match.get('distance', {})
|
||||
z_val = dist.get('longitudinal', None)
|
||||
x_val = dist.get('lateral', None)
|
||||
|
||||
# Helper: fill one bucket
|
||||
def _fill(bucket):
|
||||
bucket['lateral'].append(errs['lateral'])
|
||||
bucket['longitudinal'].append(errs['longitudinal'])
|
||||
bucket['heading'].append(errs['heading'])
|
||||
if 'longitudinal_relative' in errs:
|
||||
bucket['longitudinal_relative'].append(errs['longitudinal_relative'])
|
||||
if 'heading_relaxed' in errs:
|
||||
bucket['heading_relaxed'].append(errs['heading_relaxed'])
|
||||
if 'is_reversal' in errs:
|
||||
bucket['is_reversal'].append(errs['is_reversal'])
|
||||
bucket['samples'] += 1
|
||||
|
||||
_fill(overall[class_name])
|
||||
|
||||
# Longitudinal range bucket
|
||||
if z_val is not None:
|
||||
for lo, hi in long_ranges:
|
||||
if lo <= z_val < hi:
|
||||
_fill(by_long[class_name][_range_key_long(lo, hi)])
|
||||
break
|
||||
|
||||
# Lateral range bucket
|
||||
if x_val is not None:
|
||||
for lo, hi in lat_ranges:
|
||||
if lo <= x_val < hi:
|
||||
_fill(by_lat[class_name][_range_key_lat(lo, hi)])
|
||||
break
|
||||
|
||||
# Build result
|
||||
result = {}
|
||||
for class_name in overall:
|
||||
result[class_name] = {}
|
||||
|
||||
# overall
|
||||
if overall[class_name]['samples'] > 0:
|
||||
result[class_name]['overall'] = _finalize_class_stats(overall[class_name])
|
||||
else:
|
||||
result[class_name]['overall'] = {'num_samples': 0}
|
||||
|
||||
# per longitudinal range
|
||||
for rk, bucket in by_long[class_name].items():
|
||||
if bucket['samples'] > 0:
|
||||
result[class_name][rk] = _finalize_class_stats(bucket)
|
||||
|
||||
# per lateral range
|
||||
for rk, bucket in by_lat[class_name].items():
|
||||
if bucket['samples'] > 0:
|
||||
result[class_name][rk] = _finalize_class_stats(bucket)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def print_statistics(stats, model1_name='model1', model2_name='model2'):
|
||||
"""Print match statistics in a readable format."""
|
||||
print("\n" + "="*80)
|
||||
print("COMMON MATCH STATISTICS")
|
||||
print("="*80)
|
||||
|
||||
print(f"\nOverall:")
|
||||
print(f" {model1_name} Total Matches: {stats['model1_total']:,}")
|
||||
print(f" {model2_name} Total Matches: {stats['model2_total']:,}")
|
||||
print(f" Common Matches: {stats['common']:,} ({stats['common_percentage_of_model1']:.1f}% of {model1_name})")
|
||||
print(f" {model1_name} Unique: {stats['model1_unique']:,} ({100 - stats['common_percentage_of_model1']:.1f}%)")
|
||||
print(f" {model2_name} Unique: {stats['model2_unique']:,} ({100 - stats['common_percentage_of_model2']:.1f}%)")
|
||||
|
||||
print(f"\nPer-Class Statistics:")
|
||||
# Truncate model names if too long for column headers
|
||||
m1_short = model1_name[:10]
|
||||
m2_short = model2_name[:10]
|
||||
print(f"{'Class':<15} {m1_short:>10} {m2_short:>10} {'Common':>10} {'Common%':>10} {m1_short+' Uniq':>12} {m2_short+' Uniq':>12}")
|
||||
print("-" * 80)
|
||||
|
||||
for class_name, class_stats in sorted(stats['per_class'].items()):
|
||||
print(f"{class_name:<15} {class_stats['model1_total']:>10,} {class_stats['model2_total']:>10,} "
|
||||
f"{class_stats['common']:>10,} {class_stats['common_percentage_of_model1']:>9.1f}% "
|
||||
f"{class_stats['model1_unique']:>12,} {class_stats['model2_unique']:>12,}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Find common matches between two model evaluation results',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter
|
||||
)
|
||||
|
||||
parser.add_argument('--model1-matches', type=str, required=True,
|
||||
help='Path to model 1 detailed_3d_matches.json file')
|
||||
parser.add_argument('--model2-matches', type=str, required=True,
|
||||
help='Path to model 2 detailed_3d_matches.json file')
|
||||
parser.add_argument('--output', type=str, default='common_matches.json',
|
||||
help='Output path for common matches JSON file')
|
||||
parser.add_argument('--model1-name', type=str, default='model1',
|
||||
help='Name for model 1 (default: model1)')
|
||||
parser.add_argument('--model2-name', type=str, default='model2',
|
||||
help='Name for model 2 (default: model2)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Load detailed matches
|
||||
print(f"Loading model 1 matches from: {args.model1_matches}")
|
||||
with open(args.model1_matches, 'r') as f:
|
||||
model1_matches = json.load(f)
|
||||
|
||||
print(f"Loading model 2 matches from: {args.model2_matches}")
|
||||
with open(args.model2_matches, 'r') as f:
|
||||
model2_matches = json.load(f)
|
||||
|
||||
# Find common matches
|
||||
print("\nFinding common matches...")
|
||||
common_matches, stats = find_common_matches(model1_matches, model2_matches)
|
||||
|
||||
# Print statistics
|
||||
print_statistics(stats, args.model1_name, args.model2_name)
|
||||
|
||||
# Recompute 3D stats for common matches
|
||||
print("\nRecomputing 3D statistics for common matches...")
|
||||
model1_stats = recompute_3d_stats_from_common_matches(model1_matches, common_matches, 'model1')
|
||||
model2_stats = recompute_3d_stats_from_common_matches(model2_matches, common_matches, 'model2')
|
||||
|
||||
# Prepare output
|
||||
output_data = {
|
||||
'match_statistics': stats,
|
||||
'common_matches': common_matches,
|
||||
'model1_3d_stats': model1_stats,
|
||||
'model2_3d_stats': model2_stats,
|
||||
'model_names': {
|
||||
'model1': args.model1_name,
|
||||
'model2': args.model2_name
|
||||
}
|
||||
}
|
||||
|
||||
# Save output
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with open(output_path, 'w') as f:
|
||||
json.dump(output_data, f, indent=2)
|
||||
|
||||
print(f"\n✓ Common matches saved to: {output_path}")
|
||||
|
||||
# Print 3D stats comparison
|
||||
print("\n" + "="*80)
|
||||
print("3D STATISTICS COMPARISON (COMMON MATCHES ONLY)")
|
||||
print("="*80)
|
||||
|
||||
for class_name in sorted(model1_stats.keys()):
|
||||
if class_name not in model2_stats:
|
||||
continue
|
||||
|
||||
# New format: class stats are nested under 'overall'
|
||||
m1 = model1_stats[class_name].get('overall', model1_stats[class_name])
|
||||
m2 = model2_stats[class_name].get('overall', model2_stats[class_name])
|
||||
|
||||
print(f"\n{class_name.upper()} (n={m1.get('num_samples', 0):,}):")
|
||||
print(f"{'Metric':<20} {args.model1_name:>15} {args.model2_name:>15} {'Diff':>12} {'Change %':>10}")
|
||||
print("-" * 80)
|
||||
|
||||
for error_type in ['lateral_error', 'longitudinal_error', 'heading_error']:
|
||||
if error_type not in m1 or error_type not in m2:
|
||||
continue
|
||||
m1_mean = m1[error_type]['mean']
|
||||
m2_mean = m2[error_type]['mean']
|
||||
diff = m2_mean - m1_mean
|
||||
change_pct = (diff / m1_mean * 100) if m1_mean > 0 else 0
|
||||
|
||||
error_name = error_type.replace('_', ' ').title()
|
||||
print(f"{error_name:<20} {m1_mean:>15.4f} {m2_mean:>15.4f} {diff:>+12.4f} {change_pct:>+9.2f}%")
|
||||
|
||||
# Print relaxed heading error if available
|
||||
if 'heading_error_relaxed' in m1 and 'heading_error_relaxed' in m2:
|
||||
m1_mean = m1['heading_error_relaxed']['mean']
|
||||
m2_mean = m2['heading_error_relaxed']['mean']
|
||||
diff = m2_mean - m1_mean
|
||||
change_pct = (diff / m1_mean * 100) if m1_mean > 0 else 0
|
||||
print(f"{'Heading Error (Rlx)':<20} {m1_mean:>15.4f} {m2_mean:>15.4f} {diff:>+12.4f} {change_pct:>+9.2f}%")
|
||||
|
||||
# Print reversal statistics if available
|
||||
if 'reversal_count' in m1 and 'reversal_count' in m2:
|
||||
m1_count = m1['reversal_count']
|
||||
m1_pct = m1.get('reversal_percentage', 0)
|
||||
m2_count = m2['reversal_count']
|
||||
m2_pct = m2.get('reversal_percentage', 0)
|
||||
print(f"{'Reversals':<20} {m1_count:>11} ({m1_pct:>5.1f}%) {m2_count:>11} ({m2_pct:>5.1f}%)")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
34
eval_tools/model_comparison/gen_eval_report.sh
Executable file
34
eval_tools/model_comparison/gen_eval_report.sh
Executable file
@@ -0,0 +1,34 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# 根据 evaluation_report.json 自动生成单模型 Markdown 评测报告
|
||||
#
|
||||
# 用法:
|
||||
# bash eval_tools/model_comparison/gen_eval_report.sh <evaluation_report.json>
|
||||
# bash eval_tools/model_comparison/gen_eval_report.sh <evaluation_report.json> [--output <输出路径>] [--model "模型名"] [--date YYYY-MM-DD]
|
||||
#
|
||||
# 示例:
|
||||
# bash eval_tools/model_comparison/gen_eval_report.sh \
|
||||
# evaluation_results/.../evaluation_report.json
|
||||
#
|
||||
# bash eval_tools/model_comparison/gen_eval_report.sh \
|
||||
# evaluation_results/.../evaluation_report.json \
|
||||
# --model yolov5s-300w-newdata-cncap \
|
||||
# --date 2026-02-28
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
|
||||
|
||||
if [ $# -eq 0 ]; then
|
||||
echo "用法: bash $0 <evaluation_report.json> [选项...]"
|
||||
echo ""
|
||||
echo "选项:"
|
||||
echo " --output/-o <路径> 输出 Markdown 文件路径(默认:JSON 同目录下的 EVALUATION_REPORT.md)"
|
||||
echo " --model <名称> 模型名称(默认从目录名推断)"
|
||||
echo " --date <YYYY-MM-DD> 评测日期(默认:今天)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python "${SCRIPT_DIR}/generate_eval_report.py" "$@"
|
||||
35
eval_tools/model_comparison/gen_report.sh
Executable file
35
eval_tools/model_comparison/gen_report.sh
Executable file
@@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# 根据 comparison_report.json 自动生成 Markdown 评测报告
|
||||
#
|
||||
# 用法:
|
||||
# bash eval_tools/model_comparison/gen_report.sh <comparison_report.json>
|
||||
# bash eval_tools/model_comparison/gen_report.sh <comparison_report.json> [--output <输出路径>] [--title "标题"] [--background "背景说明"] [--date YYYY-MM-DD]
|
||||
#
|
||||
# 示例:
|
||||
# bash eval_tools/model_comparison/gen_report.sh \
|
||||
# evaluation_results/eval_results_.../comparison_common_matches_.../comparison_report.json
|
||||
#
|
||||
# bash eval_tools/model_comparison/gen_report.sh \
|
||||
# evaluation_results/.../comparison_report.json \
|
||||
# --output my_report.md \
|
||||
# --title "ROI0 模型对比报告"
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
|
||||
|
||||
if [ $# -eq 0 ]; then
|
||||
echo "用法: bash $0 <comparison_report.json> [选项...]"
|
||||
echo ""
|
||||
echo "选项:"
|
||||
echo " --output/-o <路径> 输出 Markdown 文件路径(默认:JSON 同目录下的 COMPARISON_REPORT.md)"
|
||||
echo " --title <标题> 自定义报告标题"
|
||||
echo " --background <说明> 背景说明文字"
|
||||
echo " --date <YYYY-MM-DD> 评测日期(默认:今天)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python "${SCRIPT_DIR}/generate_comparison_report.py" "$@"
|
||||
496
eval_tools/model_comparison/generate_comparison_report.py
Executable file
496
eval_tools/model_comparison/generate_comparison_report.py
Executable file
@@ -0,0 +1,496 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
自动将 comparison_report.json 转换为中文 Markdown 评测报告。
|
||||
|
||||
用法:
|
||||
python generate_comparison_report.py <comparison_report.json 路径>
|
||||
python generate_comparison_report.py <comparison_report.json 路径> --output <输出文件路径>
|
||||
python generate_comparison_report.py <comparison_report.json 路径> --title "自定义标题"
|
||||
python generate_comparison_report.py <comparison_report.json 路径> --background "背景说明文字"
|
||||
python generate_comparison_report.py <comparison_report.json 路径> --date 2026-03-01
|
||||
|
||||
示例:
|
||||
python generate_comparison_report.py \
|
||||
evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/comparison_common_matches_20260228_102849/comparison_report.json
|
||||
|
||||
python generate_comparison_report.py \
|
||||
evaluation_results/.../comparison_report.json \
|
||||
--output my_report.md \
|
||||
--title "ROI1 模型对比报告"
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
# Allow importing class_config from the eval_tools root
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
from class_config import REPORT_3D_CLASS_LABELS
|
||||
|
||||
# ── 阈值设置 ─────────────────────────────────────────────────────────────────
|
||||
# AP 差异超过此阈值才标记为"优",否则标记为"持平"
|
||||
AP_TIE_THRESHOLD = 0.005 # 0.5%
|
||||
METRIC_TIE_THRESHOLD = 0.005 # 用于 precision/recall/f1 的判断阈值(绝对值)
|
||||
ERROR_TIE_THRESHOLD_REL = 2.0 # 3D 误差相对变化(%)小于此值视为持平
|
||||
|
||||
|
||||
def fmt(v: float, decimals: int = 4) -> str:
|
||||
return f"{v:.{decimals}f}"
|
||||
|
||||
|
||||
def fmt_pct(v: float) -> str:
|
||||
sign = "+" if v >= 0 else ""
|
||||
return f"{sign}{v:.2f}%"
|
||||
|
||||
|
||||
def fmt_diff(v: float) -> str:
|
||||
sign = "+" if v >= 0 else ""
|
||||
return f"{sign}{v:.4f}"
|
||||
|
||||
|
||||
def judge(diff: float, rel: float, higher_is_better: bool = True,
|
||||
abs_thr: float = AP_TIE_THRESHOLD, rel_thr: float = None,
|
||||
model1_name: str = "model1", model2_name: str = "model2") -> str:
|
||||
"""
|
||||
根据 diff (model2 - model1) 判断哪个模型更好。
|
||||
higher_is_better=True → diff>0 代表 model2 更好
|
||||
higher_is_better=False → diff<0 代表 model2 更好(即误差更小)
|
||||
"""
|
||||
if rel_thr is not None:
|
||||
tie = abs(rel) < rel_thr
|
||||
else:
|
||||
tie = abs(diff) < abs_thr
|
||||
|
||||
if tie:
|
||||
return "⚖️ 持平"
|
||||
|
||||
m2_better = (diff > 0) if higher_is_better else (diff < 0)
|
||||
m2_short = model2_name.split("-")[-1] # e.g. "cncap"
|
||||
m1_short = model1_name.split("-")[-1] # e.g. "newdata", "mono3d"
|
||||
if m2_better:
|
||||
return f"✅ {m2_short}优"
|
||||
else:
|
||||
return f"✅ {m1_short}优"
|
||||
|
||||
|
||||
def build_report(data: dict, model1: str, model2: str,
|
||||
report_date: str, title: str = None, background: str = None) -> str:
|
||||
"""生成完整 Markdown 报告字符串。"""
|
||||
|
||||
m2d = data["2d_metrics"]
|
||||
m3d = data.get("3d_metrics", {})
|
||||
stats = data.get("match_statistics", {})
|
||||
summary = data.get("summary", {})
|
||||
|
||||
# ── 名称简写 ──────────────────────────────────────────────────────────────
|
||||
m1_short = model1
|
||||
m2_short = model2
|
||||
# 取最后一段作为简称用于表格
|
||||
m1_tag = model1.split("-")[-1] # e.g. "newdata"
|
||||
m2_tag = model2.split("-")[-1] # e.g. "cncap"
|
||||
|
||||
lines = []
|
||||
|
||||
# ── 标题 ─────────────────────────────────────────────────────────────────
|
||||
auto_title = title or f"模型对比Overall指标总结 ({model1} vs {model2} - 通用数据集评测)"
|
||||
lines.append(f"# {auto_title}")
|
||||
lines.append("")
|
||||
lines.append(f"**对比模型**: {model1} vs {model2} ")
|
||||
lines.append(f"**评测日期**: {report_date} ")
|
||||
lines.append(f"**数据集**: 通用数据集 (Common Match Cases) ")
|
||||
|
||||
total_common = stats.get("common", None)
|
||||
m1_total = stats.get("model1_total", None)
|
||||
m2_total = stats.get("model2_total", None)
|
||||
if total_common is not None and m1_total is not None and m2_total is not None:
|
||||
lines.append(f"**匹配样本**: {total_common:,} ({model1}: {m1_total:,} | {model2}: {m2_total:,})")
|
||||
else:
|
||||
lines.append(f"**匹配样本**: N/A")
|
||||
lines.append("")
|
||||
|
||||
if background:
|
||||
lines.append(f"> **背景说明**: {background}")
|
||||
else:
|
||||
lines.append(f"> **背景说明**: 本次评测对比了 {model1} 与 {model2},评估两者在通用数据集上的2D/3D检测性能差异。")
|
||||
lines.append("")
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
# ── 2D Overall ────────────────────────────────────────────────────────────
|
||||
ov = m2d["overall"]
|
||||
lines.append("## 📊 2D检测指标 (Overall)")
|
||||
lines.append("")
|
||||
lines.append("### 总体性能对比")
|
||||
lines.append("")
|
||||
lines.append(f"| 指标 | {model1} | {model2} | 差异 | 相对变化 | 结果 |")
|
||||
lines.append("|------|" + "---|" * 5)
|
||||
|
||||
def ov_row(key, label, higher_is_better=True):
|
||||
v1 = ov[key][model1]
|
||||
v2 = ov[key][model2]
|
||||
diff = ov[key]["diff"]
|
||||
rel = ov[key]["relative_change_%"]
|
||||
j = judge(diff, rel, higher_is_better, abs_thr=METRIC_TIE_THRESHOLD,
|
||||
model1_name=model1, model2_name=model2)
|
||||
return f"| **{label}** | {fmt(v1)} | {fmt(v2)} | {fmt_diff(diff)} | {fmt_pct(rel)} | {j} |"
|
||||
|
||||
lines.append(ov_row("precision", "PRECISION"))
|
||||
lines.append(ov_row("recall", "RECALL"))
|
||||
lines.append(ov_row("f1_score", "F1-Score"))
|
||||
lines.append(ov_row("map", "mAP"))
|
||||
lines.append("")
|
||||
|
||||
# 关键发现
|
||||
prec_diff = ov["precision"]["relative_change_%"]
|
||||
rec_diff = ov["recall"]["relative_change_%"]
|
||||
map_diff = ov["map"]["relative_change_%"]
|
||||
f1_diff = ov["f1_score"]["relative_change_%"]
|
||||
|
||||
ap_wins = summary.get("2d", {}).get("ap", {}).get("wins", "?")
|
||||
ap_losses = summary.get("2d", {}).get("ap", {}).get("losses", "?")
|
||||
ap_ties = summary.get("2d", {}).get("ap", {}).get("ties", "?")
|
||||
|
||||
lines.append("### 关键发现")
|
||||
lines.append("")
|
||||
lines.append(f"- 📊 **Precision**: {model2}{'领先' if prec_diff > 0 else '落后'}{fmt_pct(abs(prec_diff))},{'误检率略低' if prec_diff > 0 else '误检率略高'}")
|
||||
lines.append(f"- 📊 **Recall**: {model1 if rec_diff < 0 else model2}领先{fmt_pct(abs(rec_diff))},检出率{'更高' if rec_diff < 0 else '更低'}")
|
||||
lines.append(f"- 📊 **mAP**: {model2 if map_diff > 0 else model1}领先{fmt_pct(abs(map_diff))}({'极小差异,基本持平' if abs(map_diff) < 2 else '有一定差距'})")
|
||||
lines.append(f"- 📊 **F1-Score**: {'两模型基本持平' if abs(f1_diff) < 1 else (model2 + '更优' if f1_diff > 0 else model1 + '更优')}(差距{fmt_pct(abs(f1_diff))})")
|
||||
lines.append(f"- ⚖️ **类别赢负统计 (AP)**: {m2_tag}赢{ap_wins}类, {m1_tag}赢{ap_losses}类, 平局{ap_ties}类")
|
||||
lines.append("")
|
||||
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
# ── 2D Per-Class ──────────────────────────────────────────────────────────
|
||||
pc = m2d.get("per_class", {})
|
||||
lines.append("## 📋 2D检测指标 (Per Class)")
|
||||
lines.append("")
|
||||
lines.append("### 各类别性能对比")
|
||||
lines.append("")
|
||||
lines.append(f"| 类别 | Precision ({m1_tag}) | Precision ({m2_tag}) | Recall ({m1_tag}) | Recall ({m2_tag}) | F1 ({m1_tag}) | F1 ({m2_tag}) | AP ({m1_tag}) | AP ({m2_tag}) | AP差异 | 结果 |")
|
||||
lines.append("|------|" + "---|" * 10)
|
||||
|
||||
adv_m2 = [] # model2 明显更好的类别
|
||||
adv_m1 = [] # model1 明显更好的类别
|
||||
|
||||
for cls, cd in pc.items():
|
||||
prec1 = cd["precision"][model1]
|
||||
prec2 = cd["precision"][model2]
|
||||
rec1 = cd["recall"][model1]
|
||||
rec2 = cd["recall"][model2]
|
||||
f1_1 = cd["f1_score"][model1]
|
||||
f1_2 = cd["f1_score"][model2]
|
||||
ap1 = cd["ap"][model1]
|
||||
ap2 = cd["ap"][model2]
|
||||
ap_d = cd["ap"]["diff"]
|
||||
ap_r = cd["ap"]["relative_change_%"]
|
||||
j = judge(ap_d, ap_r, True, abs_thr=AP_TIE_THRESHOLD,
|
||||
model1_name=model1, model2_name=model2)
|
||||
lines.append(
|
||||
f"| **{cls}** | {fmt(prec1)} | {fmt(prec2)} | {fmt(rec1)} | {fmt(rec2)} "
|
||||
f"| {fmt(f1_1)} | {fmt(f1_2)} | {fmt(ap1)} | {fmt(ap2)} | {fmt_diff(ap_d)} | {j} |"
|
||||
)
|
||||
if abs(ap_r) >= 2.0: # 相对变化>=2%才算显著
|
||||
if ap_d > 0:
|
||||
adv_m2.append((cls, ap1, ap2, ap_r))
|
||||
elif ap_d < 0:
|
||||
adv_m1.append((cls, ap1, ap2, ap_r))
|
||||
|
||||
lines.append("")
|
||||
lines.append("### 类别分析")
|
||||
lines.append("")
|
||||
|
||||
if adv_m2:
|
||||
lines.append(f"**{model2} 优势类别** (AP更高):")
|
||||
for cls, ap1, ap2, rel in sorted(adv_m2, key=lambda x: -x[3]):
|
||||
mark = "**大幅领先**" if rel > 8 else "领先"
|
||||
lines.append(f"- {cls}: {m2_tag} {fmt(ap2)} > {m1_tag} {fmt(ap1)}({mark}{fmt_pct(rel)})")
|
||||
lines.append("")
|
||||
|
||||
if adv_m1:
|
||||
lines.append(f"**{model1} 优势类别** (AP更高):")
|
||||
for cls, ap1, ap2, rel in sorted(adv_m1, key=lambda x: x[3]):
|
||||
mark = "**大幅领先**" if abs(rel) > 8 else "领先"
|
||||
lines.append(f"- {cls}: {m1_tag} {fmt(ap1)} > {m2_tag} {fmt(ap2)}({mark}{fmt_pct(abs(rel))})")
|
||||
lines.append("")
|
||||
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
# ── 3D Metrics ────────────────────────────────────────────────────────────
|
||||
if m3d:
|
||||
lines.append("## 🎯 3D检测指标")
|
||||
lines.append("")
|
||||
|
||||
cls_labels = REPORT_3D_CLASS_LABELS
|
||||
|
||||
for cls_key, cls_label in cls_labels.items():
|
||||
if cls_key not in m3d:
|
||||
continue
|
||||
cd = m3d[cls_key]
|
||||
ov3 = cd.get("overall", {})
|
||||
if not ov3:
|
||||
continue
|
||||
n = cd.get("common_samples")
|
||||
n_str = f"{n:,} 个样本" if n is not None else "N/A 个样本"
|
||||
|
||||
lines.append(f"### {cls_label} - {n_str}")
|
||||
lines.append("")
|
||||
lines.append(f"| 指标 | {model1} | {model2} | 差异 | 相对变化 | 结果 |")
|
||||
lines.append("|------|" + "---|" * 5)
|
||||
|
||||
def row3d(key, label, higher_is_better=False):
|
||||
if key not in ov3:
|
||||
return None
|
||||
v1 = ov3[key][model1]["mean"]
|
||||
v2 = ov3[key][model2]["mean"]
|
||||
diff = ov3[key]["diff"]
|
||||
rel = ov3[key]["relative_change_%"]
|
||||
j = judge(diff, rel, higher_is_better,
|
||||
rel_thr=ERROR_TIE_THRESHOLD_REL,
|
||||
model1_name=model1, model2_name=model2)
|
||||
return f"| **{label}** | {fmt(v1)} | {fmt(v2)} | {fmt_diff(diff)} | {fmt_pct(rel)} | {j} |"
|
||||
|
||||
for row in [
|
||||
row3d("lateral_error", "Lateral Error"),
|
||||
row3d("longitudinal_error", "Longitudinal Error"),
|
||||
row3d("longitudinal_relative_error", "Longitudinal Relative Error"),
|
||||
row3d("heading_error", "Heading Error"),
|
||||
row3d("heading_error_relaxed", "Heading Error Relaxed"),
|
||||
]:
|
||||
if row is not None:
|
||||
lines.append(row)
|
||||
|
||||
if "reversal_info" in ov3:
|
||||
rev1 = ov3["reversal_info"][model1]
|
||||
rev2 = ov3["reversal_info"][model2]
|
||||
rev_j = "✅ " + (m2_tag if rev2["percentage"] < rev1["percentage"] else m1_tag) + "优"
|
||||
if abs(rev1["percentage"] - rev2["percentage"]) < 0.5:
|
||||
rev_j = "⚖️ 持平"
|
||||
lines.append(
|
||||
f"| **Reversal Cases** | {rev1['count']:,} ({rev1['percentage']:.2f}%) "
|
||||
f"| {rev2['count']:,} ({rev2['percentage']:.2f}%) | - | - | {rev_j} |"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# ── 纵向区间对比 ──────────────────────────────────────────────
|
||||
def _long_sort_key(k):
|
||||
stripped = k[len("long_"):].replace("m", "")
|
||||
m = re.search(r'(?<=\d)-', stripped)
|
||||
if m:
|
||||
try:
|
||||
return float(stripped[:m.start()])
|
||||
except ValueError:
|
||||
pass
|
||||
return float('inf')
|
||||
|
||||
long_keys = sorted(
|
||||
[k for k in cd.keys() if k.startswith("long_")],
|
||||
key=_long_sort_key
|
||||
)
|
||||
if long_keys:
|
||||
lines.append(f"#### 纵向区间对比")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
f"| 区间 | 样本数 "
|
||||
f"| Lat ({m1_tag}) | Lat ({m2_tag}) | Lat Δ% "
|
||||
f"| Long ({m1_tag}) | Long ({m2_tag}) | Long Δ% "
|
||||
f"| LongRel ({m1_tag}) | LongRel ({m2_tag}) | LongRel Δ% "
|
||||
f"| Head ({m1_tag}) | Head ({m2_tag}) | Head Δ% |"
|
||||
)
|
||||
lines.append("|------|" + "---|" * 13)
|
||||
for rk in long_keys:
|
||||
rb = cd[rk]
|
||||
if not rb:
|
||||
continue
|
||||
|
||||
def _rv(metric, model):
|
||||
d = rb.get(metric, {})
|
||||
if model in d:
|
||||
return fmt(d[model]["mean"])
|
||||
return "-"
|
||||
|
||||
def _rd(metric):
|
||||
d = rb.get(metric, {})
|
||||
rel = d.get("relative_change_%")
|
||||
if rel is None:
|
||||
return "-"
|
||||
return fmt_pct(rel)
|
||||
|
||||
# sample count from any available metric
|
||||
n_range = "-"
|
||||
for _mk in ("lateral_error", "longitudinal_error", "heading_error"):
|
||||
_md = rb.get(_mk, {})
|
||||
if model1 in _md and "samples" in _md[model1]:
|
||||
n_range = f"{_md[model1]['samples']:,}"
|
||||
break
|
||||
|
||||
# range label: strip prefix and trailing 'm'
|
||||
rl = rk[len("long_"):]
|
||||
|
||||
lines.append(
|
||||
f"| **{rl}** | {n_range} "
|
||||
f"| {_rv('lateral_error', model1)} | {_rv('lateral_error', model2)} | {_rd('lateral_error')} "
|
||||
f"| {_rv('longitudinal_error', model1)} | {_rv('longitudinal_error', model2)} | {_rd('longitudinal_error')} "
|
||||
f"| {_rv('longitudinal_relative_error', model1)} | {_rv('longitudinal_relative_error', model2)} | {_rd('longitudinal_relative_error')} "
|
||||
f"| {_rv('heading_error', model1)} | {_rv('heading_error', model2)} | {_rd('heading_error')} |"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
# ── Match Statistics ──────────────────────────────────────────────────────
|
||||
if stats:
|
||||
lines.append("## 📊 样本匹配统计")
|
||||
lines.append("")
|
||||
lines.append("### 整体匹配情况")
|
||||
lines.append("")
|
||||
lines.append("| 模型 | 总样本数 | 公共样本 | 独有样本 | 公共占比 |")
|
||||
lines.append("|------|----------|----------|----------|---------|")
|
||||
m1_pct = stats.get("common_percentage_of_model1", 0)
|
||||
m2_pct = stats.get("common_percentage_of_model2", 0)
|
||||
m1_uniq = stats.get("model1_unique", 0)
|
||||
m2_uniq = stats.get("model2_unique", 0)
|
||||
lines.append(f"| **{model1}** | {m1_total:,} | {total_common:,} | {m1_uniq:,} | {m1_pct:.2f}% |")
|
||||
lines.append(f"| **{model2}** | {m2_total:,} | {total_common:,} | {m2_uniq:,} | {m2_pct:.2f}% |")
|
||||
lines.append("")
|
||||
|
||||
per_cls_stats = stats.get("per_class", {})
|
||||
if per_cls_stats:
|
||||
lines.append("### 各类别匹配情况 (3D)")
|
||||
lines.append("")
|
||||
lines.append(f"| 类别 | {m1_tag}总数 | {m2_tag}总数 | 公共样本 | {m1_tag}占比 | {m2_tag}占比 |")
|
||||
lines.append("|------|" + "---|" * 5)
|
||||
for cls, cs in per_cls_stats.items():
|
||||
lines.append(
|
||||
f"| **{cls}** | {cs['model1_total']:,} | {cs['model2_total']:,} "
|
||||
f"| {cs['common']:,} | {cs['common_percentage_of_model1']:.2f}% "
|
||||
f"| {cs['common_percentage_of_model2']:.2f}% |"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
# ── Summary / Conclusions ─────────────────────────────────────────────────
|
||||
lines.append("## 🎯 结论与建议")
|
||||
lines.append("")
|
||||
lines.append("### 2D检测汇总")
|
||||
lines.append("")
|
||||
|
||||
sum2d = summary.get("2d", {})
|
||||
ap_w = sum2d.get("ap", {}).get("wins", 0)
|
||||
ap_l = sum2d.get("ap", {}).get("losses", 0)
|
||||
ap_t = sum2d.get("ap", {}).get("ties", 0)
|
||||
f1_w = sum2d.get("f1_score", {}).get("wins", 0)
|
||||
f1_l = sum2d.get("f1_score", {}).get("losses", 0)
|
||||
f1_t = sum2d.get("f1_score", {}).get("ties", 0)
|
||||
|
||||
lines.append(f"- **AP 类别统计**: {m2_tag}赢{ap_w}类 / {m1_tag}赢{ap_l}类 / 平局{ap_t}类")
|
||||
lines.append(f"- **F1 类别统计**: {m2_tag}赢{f1_w}类 / {m1_tag}赢{f1_l}类 / 平局{f1_t}类")
|
||||
lines.append(f"- **整体mAP**: {model1}={fmt(ov['map'][model1])} vs {model2}={fmt(ov['map'][model2])} ({fmt_pct(ov['map']['relative_change_%'])})")
|
||||
lines.append("")
|
||||
|
||||
if m3d:
|
||||
sum3d = summary.get("3d", {})
|
||||
lat_w = sum3d.get("lateral", {}).get("wins", 0)
|
||||
lat_l = sum3d.get("lateral", {}).get("losses", 0)
|
||||
lon_w = sum3d.get("longitudinal", {}).get("wins", 0)
|
||||
lon_l = sum3d.get("longitudinal", {}).get("losses", 0)
|
||||
hd_w = sum3d.get("heading", {}).get("wins", 0)
|
||||
hd_l = sum3d.get("heading", {}).get("losses", 0)
|
||||
|
||||
lines.append("### 3D检测汇总")
|
||||
lines.append("")
|
||||
lines.append(f"- **横向误差 (Lateral)**: {m2_tag}优{lat_w}类 / {m1_tag}优{lat_l}类")
|
||||
lines.append(f"- **纵向误差 (Longitudinal)**: {m2_tag}优{lon_w}类 / {m1_tag}优{lon_l}类")
|
||||
lines.append(f"- **航向误差 (Heading)**: {m2_tag}优{hd_w}类 / {m1_tag}优{hd_l}类")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### 综合建议")
|
||||
lines.append("")
|
||||
# 自动判断整体赢家
|
||||
map_rel = ov["map"]["relative_change_%"]
|
||||
if map_rel > 2:
|
||||
overall_winner = model2
|
||||
elif map_rel < -2:
|
||||
overall_winner = model1
|
||||
else:
|
||||
overall_winner = None
|
||||
|
||||
if overall_winner:
|
||||
lines.append(f"- 🏆 **综合mAP**: {overall_winner} 整体占优({fmt_pct(abs(map_rel))})")
|
||||
else:
|
||||
lines.append(f"- ⚖️ **综合mAP**: 两模型基本持平(差距{fmt_pct(abs(map_rel))})")
|
||||
|
||||
adv_summary_m2 = [(c, r) for c, *_, r in adv_m2]
|
||||
adv_summary_m1 = [(c, r) for c, *_, r in adv_m1]
|
||||
if adv_summary_m2:
|
||||
cls_str = "、".join(c for c, _ in adv_summary_m2)
|
||||
lines.append(f"- ✅ **{model2} 改善**: {cls_str} 类别AP有所提升")
|
||||
if adv_summary_m1:
|
||||
cls_str = "、".join(c for c, _ in adv_summary_m1)
|
||||
lines.append(f"- ⚠️ **{model2} 退化**: {cls_str} 类别AP有所下降")
|
||||
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="将 comparison_report.json 转换为中文 Markdown 评测报告"
|
||||
)
|
||||
parser.add_argument("json_path", help="comparison_report.json 的路径")
|
||||
parser.add_argument("--output", "-o", default=None,
|
||||
help="输出 Markdown 文件路径(默认与 JSON 同目录,文件名 COMPARISON_REPORT.md)")
|
||||
parser.add_argument("--title", default=None,
|
||||
help="自定义报告标题")
|
||||
parser.add_argument("--background", default=None,
|
||||
help="背景说明文字")
|
||||
parser.add_argument("--date", default=str(date.today()),
|
||||
help="评测日期 (默认今天,格式 YYYY-MM-DD)")
|
||||
args = parser.parse_args()
|
||||
|
||||
json_path = Path(args.json_path)
|
||||
if not json_path.exists():
|
||||
print(f"错误: 文件不存在: {json_path}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
with open(json_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# 自动从 JSON 中读取模型名称
|
||||
models = list(data["2d_metrics"]["overall"]["precision"].keys())
|
||||
# 过滤掉 diff / relative_change_% 等非模型 key
|
||||
skip = {"diff", "relative_change_%"}
|
||||
models = [m for m in models if m not in skip]
|
||||
if len(models) < 2:
|
||||
print("错误: 无法从 JSON 中自动识别模型名称,请检查文件格式。", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
model1, model2 = models[0], models[1]
|
||||
print(f"模型1: {model1}")
|
||||
print(f"模型2: {model2}")
|
||||
|
||||
report = build_report(data, model1, model2,
|
||||
report_date=args.date,
|
||||
title=args.title,
|
||||
background=args.background)
|
||||
|
||||
# 输出路径
|
||||
if args.output:
|
||||
out_path = Path(args.output)
|
||||
else:
|
||||
out_path = json_path.parent / "COMPARISON_REPORT.md"
|
||||
|
||||
out_path.write_text(report, encoding="utf-8")
|
||||
print(f"报告已生成: {out_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
207
eval_tools/model_comparison/generate_eval_report.py
Executable file
207
eval_tools/model_comparison/generate_eval_report.py
Executable file
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
自动将单模型 evaluation_report.json 转换为中文 Markdown 评测报告。
|
||||
|
||||
用法:
|
||||
python generate_eval_report.py <evaluation_report.json 路径>
|
||||
python generate_eval_report.py <evaluation_report.json 路径> --output <输出路径>
|
||||
python generate_eval_report.py <evaluation_report.json 路径> --model "模型名称" --date 2026-03-01
|
||||
|
||||
示例:
|
||||
python eval_tools/model_comparison/generate_eval_report.py \
|
||||
evaluation_results/.../evaluation_report.json \
|
||||
--model yolov5s-300w-newdata-cncap
|
||||
"""
|
||||
|
||||
import json
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
# Allow importing class_config from the eval_tools root
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
from class_config import REPORT_3D_CLASS_LABELS
|
||||
|
||||
|
||||
def fmt(v: float, decimals: int = 4) -> str:
|
||||
return f"{v:.{decimals}f}"
|
||||
|
||||
|
||||
def fmt2(v: float) -> str:
|
||||
return f"{v:.2f}"
|
||||
|
||||
|
||||
def pct(v: float) -> str:
|
||||
return f"{v * 100:.1f}%"
|
||||
|
||||
|
||||
def build_report(data: dict, model_name: str, report_date: str) -> str:
|
||||
lines = []
|
||||
|
||||
# ── 标题 ─────────────────────────────────────────────────────────────────
|
||||
lines.append(f"# 模型评测报告: {model_name}")
|
||||
lines.append("")
|
||||
lines.append(f"**模型**: {model_name} ")
|
||||
lines.append(f"**评测日期**: {report_date} ")
|
||||
|
||||
eval_cfg = data.get("evaluation_config", {})
|
||||
if eval_cfg:
|
||||
lines.append(f"**置信度阈值 (P/R/F1)**: {eval_cfg.get('conf_threshold', '-')} ")
|
||||
lines.append(f"**IoU 阈值**: {eval_cfg.get('iou_threshold', '-')} ")
|
||||
lines.append(f"**AP 计算方法**: {eval_cfg.get('ap_method', '-')} ")
|
||||
|
||||
lines.append("")
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
# ── 2D Overall ────────────────────────────────────────────────────────────
|
||||
ov2d = data["2d_evaluation"]["overall"]
|
||||
lines.append("## 📊 2D检测指标 (Overall)")
|
||||
lines.append("")
|
||||
lines.append("| 指标 | 数值 |")
|
||||
lines.append("|------|------|")
|
||||
lines.append(f"| **Precision** | {fmt(ov2d['precision'])} |")
|
||||
lines.append(f"| **Recall** | {fmt(ov2d['recall'])} |")
|
||||
lines.append(f"| **F1-Score** | {fmt(ov2d['f1_score'])} |")
|
||||
lines.append(f"| **mAP** | {fmt(ov2d['map'])} |")
|
||||
lines.append(f"| **TP** | {ov2d['tp']:,} |")
|
||||
lines.append(f"| **FP** | {ov2d['fp']:,} |")
|
||||
lines.append(f"| **FN** | {ov2d['fn']:,} |")
|
||||
lines.append("")
|
||||
|
||||
# ── 2D Per-Class ──────────────────────────────────────────────────────────
|
||||
lines.append("## 📋 2D检测指标 (Per Class)")
|
||||
lines.append("")
|
||||
lines.append("| 类别 | Precision | Recall | F1 | AP | GT | TP | FP | FN |")
|
||||
lines.append("|------|-----------|--------|----|----|-----|-----|-----|-----|")
|
||||
|
||||
pc2d = data["2d_evaluation"]["per_class"]
|
||||
for cls, cd in pc2d.items():
|
||||
lines.append(
|
||||
f"| **{cls}** | {fmt(cd['precision'])} | {fmt(cd['recall'])} "
|
||||
f"| {fmt(cd['f1_score'])} | {fmt(cd['ap'])} "
|
||||
f"| {cd['num_gt']:,} | {cd['tp']:,} | {cd['fp']:,} | {cd['fn']:,} |"
|
||||
)
|
||||
lines.append("")
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
# ── 3D Overall per class ──────────────────────────────────────────────────
|
||||
m3d = data.get("3d_evaluation", {})
|
||||
if not m3d:
|
||||
return "\n".join(lines)
|
||||
|
||||
lines.append("## 🎯 3D检测指标")
|
||||
lines.append("")
|
||||
|
||||
CLS_LABELS = REPORT_3D_CLASS_LABELS
|
||||
|
||||
LONG_RANGES = ["long_0-10m", "long_10-20m", "long_20-30m", "long_30-40m",
|
||||
"long_40-50m", "long_50-60m", "long_60-70m", "long_70-80m",
|
||||
"long_80-90m", "long_90-100m", "long_100-999m"]
|
||||
|
||||
for cls_key, cls_label in CLS_LABELS.items():
|
||||
if cls_key not in m3d:
|
||||
continue
|
||||
cd = m3d[cls_key]
|
||||
ov = cd["overall"]
|
||||
n = ov["num_samples"]
|
||||
|
||||
lines.append(f"### {cls_label} ({n:,} 样本)")
|
||||
lines.append("")
|
||||
|
||||
# Overall stats
|
||||
lines.append("#### 总体指标")
|
||||
lines.append("")
|
||||
lines.append("| 指标 | Mean | Median | Std | P90 |")
|
||||
lines.append("|------|------|--------|-----|-----|")
|
||||
|
||||
def stat_row(label, key):
|
||||
s = ov[key]
|
||||
return (f"| **{label}** | {fmt2(s['mean'])} | {fmt2(s['median'])} "
|
||||
f"| {fmt2(s['std'])} | {fmt2(s['percentile_90'])} |")
|
||||
|
||||
lines.append(stat_row("Lateral Error (m)", "lateral_error"))
|
||||
lines.append(stat_row("Longitudinal Error (m)", "longitudinal_error"))
|
||||
lines.append(stat_row("Long. Relative Error", "longitudinal_relative_error"))
|
||||
lines.append(stat_row("Heading Error (rad)", "heading_error"))
|
||||
lines.append(stat_row("Heading Error Relaxed", "heading_error_relaxed"))
|
||||
rev_pct = ov['reversal_percentage']
|
||||
lines.append(f"| **Reversal** | {ov['reversal_count']:,} ({rev_pct:.2f}%) | - | - | - |")
|
||||
lines.append("")
|
||||
|
||||
# Distance range breakdown
|
||||
avail_ranges = [r for r in LONG_RANGES if r in cd]
|
||||
if avail_ranges:
|
||||
lines.append("#### 按距离分段 (纵向误差 Mean / Lateral Mean / Samples)")
|
||||
lines.append("")
|
||||
lines.append("| 距离段 | 样本数 | Lateral (m) | Longitudinal (m) | Long.Rel | Heading | Reversal% |")
|
||||
lines.append("|--------|--------|-------------|------------------|----------|---------|-----------|")
|
||||
for rng in avail_ranges:
|
||||
r = cd[rng]
|
||||
rov = r
|
||||
rn = rov["num_samples"]
|
||||
if rn == 0:
|
||||
continue
|
||||
lat = rov["lateral_error"]["mean"]
|
||||
lon = rov["longitudinal_error"]["mean"]
|
||||
lrel = rov["longitudinal_relative_error"]["mean"]
|
||||
hd = rov["heading_error"]["mean"]
|
||||
rev = rov["reversal_percentage"]
|
||||
lines.append(
|
||||
f"| {rng.replace('long_', '')} | {rn:,} | {fmt2(lat)} | {fmt2(lon)} "
|
||||
f"| {lrel:.3f} | {fmt2(hd)} | {rev:.1f}% |"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="将单模型 evaluation_report.json 转换为中文 Markdown 评测报告"
|
||||
)
|
||||
parser.add_argument("json_path", help="evaluation_report.json 的路径")
|
||||
parser.add_argument("--output", "-o", default=None,
|
||||
help="输出 Markdown 文件路径(默认与 JSON 同目录,文件名 EVALUATION_REPORT.md)")
|
||||
parser.add_argument("--model", default=None,
|
||||
help="模型名称(默认从目录名推断)")
|
||||
parser.add_argument("--date", default=str(date.today()),
|
||||
help="评测日期 (默认今天,格式 YYYY-MM-DD)")
|
||||
args = parser.parse_args()
|
||||
|
||||
json_path = Path(args.json_path).resolve()
|
||||
if not json_path.exists():
|
||||
print(f"错误: 文件不存在: {json_path}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
with open(json_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# 从路径推断模型名称:取 json 所在目录的上级目录名
|
||||
if args.model:
|
||||
model_name = args.model
|
||||
else:
|
||||
# e.g. .../yolov5s-300w-newdata-cncap/20260228_102849/evaluation_report.json
|
||||
model_name = json_path.parent.parent.name
|
||||
if not model_name or model_name == ".":
|
||||
model_name = json_path.parent.name
|
||||
print(f"模型: {model_name}")
|
||||
|
||||
report = build_report(data, model_name, args.date)
|
||||
|
||||
if args.output:
|
||||
out_path = Path(args.output)
|
||||
else:
|
||||
out_path = json_path.parent / "EVALUATION_REPORT.md"
|
||||
|
||||
out_path.write_text(report, encoding="utf-8")
|
||||
print(f"报告已生成: {out_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user