单目3D初始代码

This commit is contained in:
zhao.zhu
2026-06-24 09:35:46 +08:00
commit 04a5895b6b
1153 changed files with 340700 additions and 0 deletions

View File

@@ -0,0 +1,92 @@
# Per-Case 2D Metrics Comparison Tool
This tool compares `per_case_2d` metrics between two model evaluation reports and identifies cases with significant metric differences.
## Files
- `compare_per_case_2d.py` - Main Python script for comparing per-case metrics
- `compare_per_case_2d.sh` - Shell script with pre-configured paths for mono3d vs yolov5s-300w-newdata comparison
## Usage
### Quick Start (Using Shell Script)
```bash
cd /deeplearning_team/ydong/dongying/projects/yolov5-3d
./eval_tools/model_comparison/compare_per_case_2d.sh
```
This will compare the two models and save results to `evaluation_results/per_case_2d_comparison.json`.
### Custom Comparison (Using Python Script)
```bash
python eval_tools/model_comparison/compare_per_case_2d.py \
--model1 path/to/model1/evaluation_report.json \
--model2 path/to/model2/evaluation_report.json \
--model1-name "Model-A" \
--model2-name "Model-B" \
--threshold 0.1 \
--output comparison_results.json \
--top-n 30
```
### Arguments
- `--model1`: Path to first model's evaluation_report.json (required)
- `--model2`: Path to second model's evaluation_report.json (required)
- `--model1-name`: Display name for model 1 (default: "Model-1")
- `--model2-name`: Display name for model 2 (default: "Model-2")
- `--threshold`: Threshold for significant difference, e.g., 0.1 = 10% (default: 0.1)
- `--output`: Output JSON file path (default: "per_case_comparison.json")
- `--top-n`: Number of top different cases to display (default: 20)
## Output
The script generates:
1. **Console Output**:
- Summary of total cases and common cases
- Top N cases with significant differences
- Summary statistics (mean, std, median, range) for each class and metric
2. **JSON File**: Contains detailed comparison data including:
- `summary`: Overview statistics
- `significant_differences`: List of cases exceeding the threshold
- `all_case_comparisons`: Complete per-case comparison data
- `summary_statistics`: Statistical analysis by class and metric
## Example Output
```
Top 30 Cases with Significant Differences
================================================================================
1. Case: 20251118/seq-53
Class: pedestrian, Metric: ap
mono3d: 1.0000
yolov5s-300w-newdata: 0.0000
Difference: -1.0000 (abs: 1.0000)
2. Case: 20251121/seq-30
Class: roadblock, Metric: ap
mono3d: 1.0000
yolov5s-300w-newdata: 0.0000
Difference: -1.0000 (abs: 1.0000)
...
Summary Statistics
================================================================================
VEHICLE:
ap : mean=-0.0776, std=0.1439, median=-0.0243, range=[-0.7935, +0.0994]
precision : mean=+0.1279, std=0.2248, median=+0.0934, range=[-0.9442, +0.6074]
recall : mean=-0.1210, std=0.1579, median=-0.0635, range=[-0.8975, +0.0000]
```
## Interpretation
- **Positive difference**: Model 2 performs better than Model 1
- **Negative difference**: Model 1 performs better than Model 2
- Cases are sorted by absolute difference (largest differences first)
- Summary statistics show overall trends across all cases

View File

@@ -0,0 +1,3 @@
019b8ddb-ae8a-70f3-b86f-055894c79724
019b6bf2-01a6-7029-9232-fce2bbcd2d73
019b6bf2-0124-7820-91b7-c5eb42150cd2

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,82 @@
#!/bin/bash
# Example script for comparing two model evaluation results
#
# Usage: bash eval_tools/compare_models_example.sh
# =============================================================================
# Configuration
# =============================================================================
# Model 1 (mono3d)
# Set PYTHONPATH to project root for module imports
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
MODEL1_REPORT="eval_results_multiprocess/mono3d/20260203_162537/evaluation_report.json"
MODEL1_NAME="mono3d"
# Model 2 (yolov5s-300w)
MODEL2_REPORT="eval_results_multiprocess/yolov5s/20260203_161644/evaluation_report.json"
MODEL2_NAME="yolov5s-300w"
# Output directory
OUTPUT_DIR="comparison_results/$(date +%Y%m%d_%H%M%S)"
# =============================================================================
# Run Comparison
# =============================================================================
echo "=========================================="
echo "Model Evaluation Comparison"
echo "=========================================="
echo "Model 1: $MODEL1_NAME"
echo " Report: $MODEL1_REPORT"
echo "Model 2: $MODEL2_NAME"
echo " Report: $MODEL2_REPORT"
echo "Output: $OUTPUT_DIR"
echo "=========================================="
# Check if reports exist
if [ ! -f "$MODEL1_REPORT" ]; then
echo "Error: Model 1 report not found: $MODEL1_REPORT"
exit 1
fi
if [ ! -f "$MODEL2_REPORT" ]; then
echo "Error: Model 2 report not found: $MODEL2_REPORT"
exit 1
fi
# Run comparison with visualization
python eval_tools/model_comparison/compare_models_visualize.py \
--model1 "$MODEL1_REPORT" \
--model2 "$MODEL2_REPORT" \
--output-dir "$OUTPUT_DIR" \
--model1-name "$MODEL1_NAME" \
--model2-name "$MODEL2_NAME"
# Check if comparison was successful
if [ $? -eq 0 ]; then
echo ""
echo "✓ Comparison completed successfully!"
echo ""
echo "View results:"
echo " Text report: $OUTPUT_DIR/comparison_report.txt"
echo " JSON report: $OUTPUT_DIR/comparison_report.json"
echo " Plots: $OUTPUT_DIR/comparison_*.png"
echo ""
# Display summary from text report
if [ -f "$OUTPUT_DIR/comparison_report.txt" ]; then
echo "=========================================="
echo "Quick Summary (from report):"
echo "=========================================="
grep -A 20 "2D DETECTION METRICS - OVERALL COMPARISON" "$OUTPUT_DIR/comparison_report.txt" | head -25
fi
else
echo ""
echo "✗ Comparison failed!"
exit 1
fi

View File

@@ -0,0 +1,48 @@
#!/bin/bash
#
# Compare Models Only (Skip Evaluation Steps)
#
# This script runs only the comparison steps (Step 4-5), assuming that
# evaluation and common match finding have already been completed.
#
# Usage:
# bash eval_tools/compare_models_only.sh <MODEL1_DIR> <MODEL2_DIR> <COMMON_MATCHES_JSON>
#
# Example:
# bash eval_tools/compare_models_only.sh \
# eval_results_common_match_comparison/mono3d/20260203_210259 \
# eval_results_common_match_comparison/yolov5s-300w/20260203_210259 \
# eval_results_common_match_comparison/common_matches_20260203_210259/common_matches.json
set -e # Exit on error
# Set PYTHONPATH to project root for module imports
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
MODEL1_DIR="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/yolov5s-300w-newdata/20260228_102849"
MODEL2_DIR="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/yolov5s-300w-newdata-cncap/20260228_102849"
MODEL1_NAME="yolov5s-300w-newdata"
MODEL2_NAME="yolov5s-300w-newdata-cncap"
COMMON_MATCHES_DIR="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/common_matches_20260228_102849"
COMPARISON_DIR="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/comparison_common_matches_20260228_102849-v2"
python eval_tools/model_comparison/compare_models.py \
--model1 ${MODEL1_DIR}/evaluation_report.json \
--model2 ${MODEL2_DIR}/evaluation_report.json \
--model1-name "${MODEL1_NAME}" \
--model2-name "${MODEL2_NAME}" \
--common-matches ${COMMON_MATCHES_DIR}/common_matches.json \
--output-dir ${COMPARISON_DIR}
echo "✓ Comparison results saved to: ${COMPARISON_DIR}"
# python eval_tools/model_comparison/compare_models.py \
# --model1 ${MODEL1_DIR}/evaluation_report.json \
# --model2 ${MODEL2_DIR}/evaluation_report.json \
# --model1-name "${MODEL1_NAME}" \
# --model2-name "${MODEL2_NAME}" \
# --output-dir ${COMPARISON_TRADITIONAL_DIR}

View File

@@ -0,0 +1,374 @@
#!/usr/bin/env python3
"""
Model Evaluation Comparison Tool with Visualization
Extended version with plotting capabilities.
Usage:
python eval_tools/compare_models_visualize.py \
--model1 eval_results/model1/evaluation_report.json \
--model2 eval_results/model2/evaluation_report.json \
--output-dir comparison_results \
--model1-name "mono3d" \
--model2-name "yolov5s-300w"
"""
import argparse
import json
import os
import sys
from pathlib import Path
# Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent.parent))
from eval_tools.compare_models import ModelComparator
try:
import matplotlib
matplotlib.use('Agg') # Use non-interactive backend
import matplotlib.pyplot as plt
import numpy as np
MATPLOTLIB_AVAILABLE = True
except ImportError:
MATPLOTLIB_AVAILABLE = False
print("Warning: matplotlib not available, visualization will be skipped")
class VisualizationComparator(ModelComparator):
"""Extended comparator with visualization capabilities."""
def plot_2d_metrics_comparison(self, output_dir):
"""Plot 2D metrics comparison."""
if not MATPLOTLIB_AVAILABLE:
print("Skipping 2D metrics plot (matplotlib not available)")
return
print("\nGenerating 2D metrics comparison plots...")
comparison = self.comparison_results.get('2d_metrics', {})
if not comparison:
return
# Overall metrics bar chart
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
fig.suptitle('2D Detection Metrics - Overall Comparison', fontsize=16, fontweight='bold')
overall = comparison['overall']
metrics = ['precision', 'recall', 'map']
metric_names = ['Precision', 'Recall', 'mAP']
for idx, (metric, metric_name) in enumerate(zip(metrics, metric_names)):
if metric not in overall:
continue
values = overall[metric]
model_names = [self.model1_name, self.model2_name]
model_values = [values[self.model1_name], values[self.model2_name]]
bars = axes[idx].bar(model_names, model_values, color=['#3498db', '#e74c3c'])
axes[idx].set_ylabel(metric_name)
axes[idx].set_title(f'{metric_name}')
axes[idx].set_ylim(0, 1.0)
axes[idx].grid(axis='y', alpha=0.3)
# Add value labels on bars
for bar in bars:
height = bar.get_height()
axes[idx].text(bar.get_x() + bar.get_width()/2., height,
f'{height:.3f}',
ha='center', va='bottom', fontsize=10)
plt.tight_layout()
output_file = os.path.join(output_dir, 'comparison_2d_overall.png')
plt.savefig(output_file, dpi=150, bbox_inches='tight')
plt.close()
print(f" ✓ Saved: {output_file}")
# Per-class AP comparison
per_class = comparison.get('per_class', {})
if per_class:
class_names = sorted(per_class.keys())
m1_aps = [per_class[c]['ap'][self.model1_name] for c in class_names]
m2_aps = [per_class[c]['ap'][self.model2_name] for c in class_names]
fig, ax = plt.subplots(figsize=(12, 6))
x = np.arange(len(class_names))
width = 0.35
bars1 = ax.bar(x - width/2, m1_aps, width, label=self.model1_name, color='#3498db')
bars2 = ax.bar(x + width/2, m2_aps, width, label=self.model2_name, color='#e74c3c')
ax.set_xlabel('Class', fontsize=12)
ax.set_ylabel('Average Precision (AP)', fontsize=12)
ax.set_title('Per-Class AP Comparison', fontsize=14, fontweight='bold')
ax.set_xticks(x)
ax.set_xticklabels(class_names, rotation=45, ha='right')
ax.legend()
ax.grid(axis='y', alpha=0.3)
plt.tight_layout()
output_file = os.path.join(output_dir, 'comparison_2d_per_class.png')
plt.savefig(output_file, dpi=150, bbox_inches='tight')
plt.close()
print(f" ✓ Saved: {output_file}")
def plot_3d_metrics_comparison(self, output_dir):
"""Plot 3D metrics comparison."""
if not MATPLOTLIB_AVAILABLE:
print("Skipping 3D metrics plot (matplotlib not available)")
return
print("\nGenerating 3D metrics comparison plots...")
comparison = self.comparison_results.get('3d_metrics', {})
if not comparison:
return
# Sort distance ranges by starting distance value
def get_range_start(range_key):
if range_key == 'overall':
return -1 # Put 'overall' at the beginning
try:
# Extract starting distance from format like "0-20m" or "100-999m"
return int(range_key.split('-')[0])
except (ValueError, IndexError):
return float('inf')
# For each class with distance ranges
for class_name, ranges in comparison.items():
if not ranges:
continue
# Check if we have distance ranges, sorted by distance
range_keys = sorted([k for k in ranges.keys() if k != 'overall'], key=get_range_start)
if not range_keys:
range_keys = ['overall']
# Create subplots for lateral, longitudinal, and heading errors
fig, axes = plt.subplots(1, 3, figsize=(18, 5))
fig.suptitle(f'3D Detection Metrics - {class_name.upper()}', fontsize=16, fontweight='bold')
error_types = ['lateral_error', 'longitudinal_error', 'heading_error']
error_names = ['Lateral Error (m)', 'Longitudinal Error (m)', 'Heading Error (rad)']
for idx, (error_type, error_name) in enumerate(zip(error_types, error_names)):
m1_values = []
m2_values = []
m1_stds = []
m2_stds = []
labels = []
for range_key in range_keys:
if range_key not in ranges:
continue
metrics = ranges[range_key]
if error_type not in metrics:
continue
data = metrics[error_type]
m1_values.append(data[self.model1_name]['mean'])
m2_values.append(data[self.model2_name]['mean'])
m1_stds.append(data[self.model1_name]['std'])
m2_stds.append(data[self.model2_name]['std'])
labels.append(range_key)
if not m1_values:
continue
x = np.arange(len(labels))
width = 0.35
bars1 = axes[idx].bar(x - width/2, m1_values, width,
yerr=m1_stds, label=self.model1_name,
color='#3498db', alpha=0.8, capsize=5)
bars2 = axes[idx].bar(x + width/2, m2_values, width,
yerr=m2_stds, label=self.model2_name,
color='#e74c3c', alpha=0.8, capsize=5)
axes[idx].set_xlabel('Distance Range', fontsize=10)
axes[idx].set_ylabel(error_name, fontsize=10)
axes[idx].set_title(error_name.split('(')[0], fontsize=12)
axes[idx].set_xticks(x)
axes[idx].set_xticklabels(labels, rotation=45, ha='right')
axes[idx].legend()
axes[idx].grid(axis='y', alpha=0.3)
plt.tight_layout()
output_file = os.path.join(output_dir, f'comparison_3d_{class_name}.png')
plt.savefig(output_file, dpi=150, bbox_inches='tight')
plt.close()
print(f" ✓ Saved: {output_file}")
def plot_improvement_heatmap(self, output_dir):
"""Plot improvement heatmap for 3D metrics."""
if not MATPLOTLIB_AVAILABLE:
print("Skipping improvement heatmap (matplotlib not available)")
return
print("\nGenerating improvement heatmap...")
comparison = self.comparison_results.get('3d_metrics', {})
if not comparison:
return
# Collect improvement data
data_matrix = []
row_labels = []
col_labels = ['Lateral', 'Longitudinal', 'Heading']
# Sort distance ranges by starting distance value
def get_range_start(range_key):
if range_key == 'overall':
return -1 # Put 'overall' at the beginning of each class
try:
# Extract starting distance from format like "0-20m" or "100-999m"
return int(range_key.split('-')[0])
except (ValueError, IndexError):
return float('inf')
for class_name, ranges in sorted(comparison.items()):
# Sort ranges: overall first, then by distance
sorted_range_keys = sorted(ranges.keys(), key=get_range_start)
for range_key in sorted_range_keys:
metrics = ranges[range_key]
row_data = []
for error_type in ['lateral_error', 'longitudinal_error', 'heading_error']:
if error_type in metrics:
# Negative change % means improvement (lower error)
change = -metrics[error_type]['relative_change_%']
row_data.append(change)
else:
row_data.append(0)
if any(x != 0 for x in row_data):
data_matrix.append(row_data)
label = f"{class_name}\n{range_key}"
row_labels.append(label)
if not data_matrix:
return
data_matrix = np.array(data_matrix)
fig, ax = plt.subplots(figsize=(10, max(6, len(row_labels) * 0.5)))
# Create heatmap
im = ax.imshow(data_matrix, cmap='RdYlGn', aspect='auto', vmin=-50, vmax=50)
# Set ticks
ax.set_xticks(np.arange(len(col_labels)))
ax.set_yticks(np.arange(len(row_labels)))
ax.set_xticklabels(col_labels)
ax.set_yticklabels(row_labels, fontsize=8)
# Add colorbar
cbar = plt.colorbar(im, ax=ax)
cbar.set_label(f'Improvement % ({self.model2_name} vs {self.model1_name})', rotation=270, labelpad=20)
# Add text annotations
for i in range(len(row_labels)):
for j in range(len(col_labels)):
text = ax.text(j, i, f'{data_matrix[i, j]:.1f}%',
ha="center", va="center", color="black", fontsize=8)
ax.set_title(f'3D Metrics Improvement Heatmap\n(Positive = {self.model2_name} Better)',
fontsize=14, fontweight='bold')
plt.tight_layout()
output_file = os.path.join(output_dir, 'comparison_3d_improvement_heatmap.png')
plt.savefig(output_file, dpi=150, bbox_inches='tight')
plt.close()
print(f" ✓ Saved: {output_file}")
def generate_visualizations(self, output_dir):
"""Generate all visualizations."""
if not MATPLOTLIB_AVAILABLE:
print("\n⚠ Matplotlib not available, skipping visualizations")
print("Install with: pip install matplotlib")
return
print("\n" + "="*80)
print("GENERATING VISUALIZATIONS")
print("="*80)
self.plot_2d_metrics_comparison(output_dir)
self.plot_3d_metrics_comparison(output_dir)
self.plot_improvement_heatmap(output_dir)
print("\n✓ All visualizations generated")
def main():
"""Main function."""
parser = argparse.ArgumentParser(
description='Compare evaluation results from two models with visualization',
formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument('--model1', type=str, required=True,
help='Path to model 1 evaluation report JSON')
parser.add_argument('--model2', type=str, required=True,
help='Path to model 2 evaluation report JSON')
parser.add_argument('--output-dir', type=str, default='comparison_results',
help='Output directory for comparison results')
parser.add_argument('--model1-name', type=str, default='Model-1',
help='Display name for model 1')
parser.add_argument('--model2-name', type=str, default='Model-2',
help='Display name for model 2')
parser.add_argument('--no-plots', action='store_true',
help='Skip visualization generation')
args = parser.parse_args()
# Load reports
print("="*80)
print("MODEL COMPARISON TOOL (with Visualization)")
print("="*80)
print(f"\nLoading model 1: {args.model1}")
with open(args.model1, 'r') as f:
model1_report = json.load(f)
print(f"Loading model 2: {args.model2}")
with open(args.model2, 'r') as f:
model2_report = json.load(f)
# Create output directory
os.makedirs(args.output_dir, exist_ok=True)
# Compare models
comparator = VisualizationComparator(
model1_report,
model2_report,
model1_name=args.model1_name,
model2_name=args.model2_name
)
results = comparator.compare_all()
# Generate reports
text_output = os.path.join(args.output_dir, 'comparison_report.txt')
json_output = os.path.join(args.output_dir, 'comparison_report.json')
comparator.generate_text_report(text_output)
comparator.generate_json_report(json_output)
# Generate visualizations
if not args.no_plots:
comparator.generate_visualizations(args.output_dir)
print("\n" + "="*80)
print("COMPARISON COMPLETE")
print("="*80)
print(f"\nResults saved to: {args.output_dir}/")
print(f" - Text report: comparison_report.txt")
print(f" - JSON report: comparison_report.json")
if not args.no_plots and MATPLOTLIB_AVAILABLE:
print(f" - Visualization plots: comparison_*.png")
print("")
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,185 @@
#!/bin/bash
#
# Common Match Comparison Workflow Example
#
# This script demonstrates the complete workflow for comparing two models
# using only the GT objects that both models successfully matched.
#
# Usage:
# bash eval_tools/compare_models_with_common_matches.sh
set -e # Exit on error
# Set PYTHONPATH to project root for module imports
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
echo "================================================================================"
echo "Common Match Comparison Workflow"
echo "================================================================================"
# Configuration
MODEL1_CONFIG="eval_tools/configs/eval_config_yolov5s_cncap_768-roi1.yaml"
MODEL2_CONFIG="eval_tools/configs/eval_config_yolov5s_cncap-roi1.yaml"
OUTPUT_BASE="evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_768_roi1-conf0.4-v2"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
MODEL1_NAME="20260317"
MODEL2_NAME="20260228"
# Heading tolerance mode: strict, relaxed, or both
HEADING_TOLERANCE="both"
# Step 1: Evaluate Model 1 with detailed match saving
echo ""
echo "Step 1: Evaluating Model 1 (${MODEL1_NAME}) with detailed match saving..."
echo " Heading tolerance: ${HEADING_TOLERANCE}"
echo "--------------------------------------------------------------------------------"
MODEL1_OUTPUT="${OUTPUT_BASE}/${MODEL1_NAME}/${TIMESTAMP}"
python eval_tools/core/eval.py \
--config ${MODEL1_CONFIG} \
--output-dir ${MODEL1_OUTPUT} \
--heading-tolerance ${HEADING_TOLERANCE} \
--save-detailed-matches
# Use the output directory we specified
MODEL1_DIR=${MODEL1_OUTPUT}
if [ ! -d "$MODEL1_DIR" ]; then
echo "Error: Could not find Model 1 evaluation results at ${MODEL1_DIR}"
exit 1
fi
echo "✓ Model 1 results: ${MODEL1_DIR}"
# Generate Markdown report for Model 1
python ${SCRIPT_DIR}/generate_eval_report.py \
${MODEL1_DIR}/evaluation_report.json \
--model "${MODEL1_NAME}" \
--date $(date +%Y-%m-%d)
echo "✓ Model 1 Markdown report: ${MODEL1_DIR}/EVALUATION_REPORT.md"
echo ""
echo "Step 2: Evaluating Model 2 (${MODEL2_NAME}) with detailed match saving..."
echo " Heading tolerance: ${HEADING_TOLERANCE}"
echo "--------------------------------------------------------------------------------"
MODEL2_OUTPUT="${OUTPUT_BASE}/${MODEL2_NAME}/${TIMESTAMP}"
python eval_tools/core/eval.py \
--config ${MODEL2_CONFIG} \
--output-dir ${MODEL2_OUTPUT} \
--heading-tolerance ${HEADING_TOLERANCE} \
--save-detailed-matches
# Use the output directory we specified
MODEL2_DIR=${MODEL2_OUTPUT}
if [ ! -d "$MODEL2_DIR" ]; then
echo "Error: Could not find Model 2 evaluation results at ${MODEL2_DIR}"
exit 1
fi
echo "✓ Model 2 results: ${MODEL2_DIR}"
# Generate Markdown report for Model 2
python ${SCRIPT_DIR}/generate_eval_report.py \
${MODEL2_DIR}/evaluation_report.json \
--model "${MODEL2_NAME}" \
--date $(date +%Y-%m-%d)
echo "✓ Model 2 Markdown report: ${MODEL2_DIR}/EVALUATION_REPORT.md"
# Step 3: Find common matches
echo ""
echo "Step 3: Finding common matches between the two models..."
echo "--------------------------------------------------------------------------------"
COMMON_MATCHES_DIR="${OUTPUT_BASE}/common_matches_${TIMESTAMP}"
mkdir -p ${COMMON_MATCHES_DIR}
python eval_tools/model_comparison/find_common_matches.py \
--model1-matches ${MODEL1_DIR}/detailed_3d_matches.json \
--model2-matches ${MODEL2_DIR}/detailed_3d_matches.json \
--output ${COMMON_MATCHES_DIR}/common_matches.json \
--model1-name "${MODEL1_NAME}" \
--model2-name "${MODEL2_NAME}"
echo "✓ Common matches saved to: ${COMMON_MATCHES_DIR}/common_matches.json"
# Step 4: Compare models using common matches
echo ""
echo "Step 4: Comparing models using common matches only..."
echo "--------------------------------------------------------------------------------"
COMPARISON_DIR="${OUTPUT_BASE}/comparison_common_matches_${TIMESTAMP}"
python eval_tools/model_comparison/compare_models.py \
--model1 ${MODEL1_DIR}/evaluation_report.json \
--model2 ${MODEL2_DIR}/evaluation_report.json \
--model1-name "${MODEL1_NAME}" \
--model2-name "${MODEL2_NAME}" \
--common-matches ${COMMON_MATCHES_DIR}/common_matches.json \
--output-dir ${COMPARISON_DIR}
echo "✓ Comparison results saved to: ${COMPARISON_DIR}"
# Generate Markdown report for common-match comparison
python ${SCRIPT_DIR}/generate_comparison_report.py \
${COMPARISON_DIR}/comparison_report.json \
--date $(date +%Y-%m-%d)
echo "✓ Markdown report: ${COMPARISON_DIR}/COMPARISON_REPORT.md"
# Step 5: Also run traditional comparison (without common match filtering)
echo ""
echo "Step 5: Running traditional comparison (all matches) for reference..."
echo "--------------------------------------------------------------------------------"
COMPARISON_TRADITIONAL_DIR="${OUTPUT_BASE}/comparison_all_matches_${TIMESTAMP}"
python eval_tools/model_comparison/compare_models.py \
--model1 ${MODEL1_DIR}/evaluation_report.json \
--model2 ${MODEL2_DIR}/evaluation_report.json \
--model1-name "${MODEL1_NAME}" \
--model2-name "${MODEL2_NAME}" \
--output-dir ${COMPARISON_TRADITIONAL_DIR}
echo "✓ Traditional comparison saved to: ${COMPARISON_TRADITIONAL_DIR}"
# Generate Markdown report for traditional comparison
python ${SCRIPT_DIR}/generate_comparison_report.py \
${COMPARISON_TRADITIONAL_DIR}/comparison_report.json \
--date $(date +%Y-%m-%d)
echo "✓ Markdown report: ${COMPARISON_TRADITIONAL_DIR}/COMPARISON_REPORT.md"
# Summary
echo ""
echo "================================================================================"
echo "WORKFLOW COMPLETE!"
echo "================================================================================"
echo ""
echo "Results Summary:"
echo " Model 1 (${MODEL1_NAME}):"
echo " - Evaluation: ${MODEL1_DIR}"
echo " - Detailed matches: ${MODEL1_DIR}/detailed_3d_matches.json"
echo " - Markdown report: ${MODEL1_DIR}/EVALUATION_REPORT.md"
echo ""
echo " Model 2 (${MODEL2_NAME}):"
echo " - Evaluation: ${MODEL2_DIR}"
echo " - Detailed matches: ${MODEL2_DIR}/detailed_3d_matches.json"
echo " - Markdown report: ${MODEL2_DIR}/EVALUATION_REPORT.md"
echo ""
echo " Common Matches Analysis:"
echo " - Common matches data: ${COMMON_MATCHES_DIR}/common_matches.json"
echo ""
echo " Comparison Results:"
echo " - Common matches only: ${COMPARISON_DIR}/"
echo " - All matches (traditional): ${COMPARISON_TRADITIONAL_DIR}/"
echo ""
echo "Key Files to Review:"
echo " 1. ${COMPARISON_DIR}/comparison_report.txt"
echo " (3D comparison based on common matches - fair comparison)"
echo " 1b. ${COMPARISON_DIR}/COMPARISON_REPORT.md"
echo " (Markdown report - common matches)"
echo ""
echo " 2. ${COMPARISON_TRADITIONAL_DIR}/comparison_report.txt"
echo " (Traditional comparison with all matches - for reference)"
echo " 2b. ${COMPARISON_TRADITIONAL_DIR}/COMPARISON_REPORT.md"
echo " (Markdown report - all matches)"
echo ""
echo " 3. ${COMMON_MATCHES_DIR}/common_matches.json"
echo " (Detailed statistics about match differences)"
echo ""
echo "To view the common-match comparison report:"
echo " cat ${COMPARISON_DIR}/comparison_report.txt"
echo ""

View File

@@ -0,0 +1,285 @@
#!/usr/bin/env python3
"""
Per-Case 2D Metrics Comparison Tool
This script compares per_case_2d metrics from two model evaluation reports
and identifies cases with significant metric differences.
Usage:
python eval_tools/model_comparison/compare_per_case_2d.py \
--model1 evaluation_results/.../evaluation_report.json \
--model2 evaluation_results/.../evaluation_report.json \
--threshold 0.1 \
--output comparison_per_case_2d.json
Example:
python eval_tools/model_comparison/compare_per_case_2d.py \
--model1 evaluation_results/eval_results_common_match_comparison_CNCAP_roi0/mono3d/20260211_113153/evaluation_report.json \
--model2 evaluation_results/eval_results_common_match_comparison_CNCAP_roi0/yolov5s-300w-newdata/20260211_113153/evaluation_report.json \
--threshold 0.1 \
--output per_case_comparison.json
"""
import argparse
import json
import os
from pathlib import Path
from collections import defaultdict
import sys
import numpy as np
# Allow importing class_config from the eval_tools root
sys.path.insert(0, str(Path(__file__).parent.parent))
from class_config import CLASS_NAMES
class PerCaseComparator:
"""Compare per_case_2d metrics between two models."""
def __init__(self, model1_report, model2_report, model1_name="Model-1", model2_name="Model-2"):
"""
Initialize comparator.
Args:
model1_report: dict, evaluation report for model 1
model2_report: dict, evaluation report for model 2
model1_name: str, display name for model 1
model2_name: str, display name for model 2
"""
self.model1_report = model1_report
self.model2_report = model2_report
self.model1_name = model1_name
self.model2_name = model2_name
def compare_per_case_metrics(self, threshold=0.1, metric_name='ap'):
"""
Compare per_case_2d metrics and identify cases with significant differences.
Args:
threshold: float, threshold for significant difference (default 0.1 = 10%)
metric_name: str, metric to compare ('ap', 'precision', 'recall')
Returns:
dict with comparison results
"""
print(f"\n{'='*80}")
print(f"Comparing Per-Case 2D Metrics (threshold={threshold*100}%)")
print(f"{'='*80}\n")
# Get per_case_2d data
m1_cases = self.model1_report.get('per_case_2d', {})
m2_cases = self.model2_report.get('per_case_2d', {})
# Find common cases
common_cases = set(m1_cases.keys()) & set(m2_cases.keys())
print(f"Total cases in {self.model1_name}: {len(m1_cases)}")
print(f"Total cases in {self.model2_name}: {len(m2_cases)}")
print(f"Common cases: {len(common_cases)}\n")
# Compare each case
case_comparisons = {}
significant_diffs = []
for case_name in sorted(common_cases):
m1_case = m1_cases[case_name]
m2_case = m2_cases[case_name]
case_comp = {
'case_name': case_name,
'per_class': {},
'max_diff': 0.0,
'max_diff_class': None,
'max_diff_metric': None
}
# Compare per-class metrics
m1_classes = m1_case.get('per_class', {})
m2_classes = m2_case.get('per_class', {})
for class_name in m1_classes.keys():
if class_name not in m2_classes:
continue
m1_class = m1_classes[class_name]
m2_class = m2_classes[class_name]
class_comp = {}
for metric in ['precision', 'recall', 'ap']:
m1_val = m1_class.get(metric, 0.0)
m2_val = m2_class.get(metric, 0.0)
diff = m2_val - m1_val
class_comp[metric] = {
self.model1_name: m1_val,
self.model2_name: m2_val,
'diff': diff,
'abs_diff': abs(diff)
}
# Track maximum difference
if abs(diff) > case_comp['max_diff']:
case_comp['max_diff'] = abs(diff)
case_comp['max_diff_class'] = class_name
case_comp['max_diff_metric'] = metric
# Add count metrics for context
class_comp['counts'] = {
'num_gt': m1_class.get('num_gt', 0),
'num_det_m1': m1_class.get('num_det', 0),
'num_det_m2': m2_class.get('num_det', 0),
}
case_comp['per_class'][class_name] = class_comp
case_comparisons[case_name] = case_comp
# Check if this case has significant differences
if case_comp['max_diff'] >= threshold:
significant_diffs.append({
'case_name': case_name,
'max_diff': case_comp['max_diff'],
'class': case_comp['max_diff_class'],
'metric': case_comp['max_diff_metric'],
'details': case_comp['per_class'][case_comp['max_diff_class']][case_comp['max_diff_metric']]
})
# Sort by maximum difference
significant_diffs.sort(key=lambda x: x['max_diff'], reverse=True)
results = {
'summary': {
'total_common_cases': len(common_cases),
'cases_with_significant_diff': len(significant_diffs),
'threshold': threshold,
'model1_name': self.model1_name,
'model2_name': self.model2_name
},
'significant_differences': significant_diffs,
'all_case_comparisons': case_comparisons
}
return results
def print_significant_differences(self, results, top_n=20):
"""Print top N cases with significant differences."""
sig_diffs = results['significant_differences']
print(f"\n{'='*80}")
print(f"Top {min(top_n, len(sig_diffs))} Cases with Significant Differences")
print(f"{'='*80}\n")
for i, diff in enumerate(sig_diffs[:top_n], 1):
details = diff['details']
print(f"{i}. Case: {diff['case_name']}")
print(f" Class: {diff['class']}, Metric: {diff['metric']}")
print(f" {self.model1_name}: {details[self.model1_name]:.4f}")
print(f" {self.model2_name}: {details[self.model2_name]:.4f}")
print(f" Difference: {details['diff']:+.4f} (abs: {diff['max_diff']:.4f})")
print()
def generate_summary_stats(self, results):
"""Generate summary statistics."""
all_comps = results['all_case_comparisons']
# Collect all differences by class and metric
diffs_by_class_metric = defaultdict(list)
for case_name, case_comp in all_comps.items():
for class_name, class_comp in case_comp['per_class'].items():
for metric in ['precision', 'recall', 'ap']:
diff = class_comp[metric]['diff']
diffs_by_class_metric[(class_name, metric)].append(diff)
# Calculate statistics
stats = {}
for (class_name, metric), diffs in diffs_by_class_metric.items():
diffs_array = np.array(diffs)
stats[f"{class_name}_{metric}"] = {
'mean_diff': float(np.mean(diffs_array)),
'std_diff': float(np.std(diffs_array)),
'median_diff': float(np.median(diffs_array)),
'min_diff': float(np.min(diffs_array)),
'max_diff': float(np.max(diffs_array)),
'num_cases': len(diffs)
}
return stats
def main():
parser = argparse.ArgumentParser(
description='Compare per_case_2d metrics between two model evaluation reports'
)
parser.add_argument('--model1', type=str, required=True,
help='Path to model 1 evaluation_report.json')
parser.add_argument('--model2', type=str, required=True,
help='Path to model 2 evaluation_report.json')
parser.add_argument('--model1-name', type=str, default='Model-1',
help='Display name for model 1')
parser.add_argument('--model2-name', type=str, default='Model-2',
help='Display name for model 2')
parser.add_argument('--threshold', type=float, default=0.1,
help='Threshold for significant difference (default: 0.1 = 10%%)')
parser.add_argument('--output', type=str, default='per_case_comparison.json',
help='Output JSON file path')
parser.add_argument('--top-n', type=int, default=20,
help='Number of top different cases to display (default: 20)')
args = parser.parse_args()
# Load evaluation reports
print(f"Loading {args.model1}...")
with open(args.model1, 'r') as f:
model1_report = json.load(f)
print(f"Loading {args.model2}...")
with open(args.model2, 'r') as f:
model2_report = json.load(f)
# Create comparator
comparator = PerCaseComparator(
model1_report, model2_report,
model1_name=args.model1_name,
model2_name=args.model2_name
)
# Compare metrics
results = comparator.compare_per_case_metrics(threshold=args.threshold)
# Print significant differences
comparator.print_significant_differences(results, top_n=args.top_n)
# Generate summary statistics
print(f"\n{'='*80}")
print("Summary Statistics")
print(f"{'='*80}\n")
stats = comparator.generate_summary_stats(results)
# Print stats for main classes (all 3D classes, skipping vehicle sub-buckets)
for class_name in [n for n in CLASS_NAMES.values() if n in stats or f"{n}_ap" in stats]:
print(f"\n{class_name.upper()}:")
for metric in ['ap', 'precision', 'recall']:
key = f"{class_name}_{metric}"
if key in stats:
s = stats[key]
print(f" {metric:10s}: mean={s['mean_diff']:+.4f}, "
f"std={s['std_diff']:.4f}, median={s['median_diff']:+.4f}, "
f"range=[{s['min_diff']:+.4f}, {s['max_diff']:+.4f}]")
# Save results
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
# Add summary stats to results
results['summary_statistics'] = stats
with open(output_path, 'w') as f:
json.dump(results, f, indent=2)
print(f"\n{'='*80}")
print(f"Results saved to: {output_path}")
print(f"{'='*80}\n")
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,26 @@
#!/bin/bash
# Compare per_case_2d metrics between mono3d and yolov5s-300w-newdata models
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
MODEL1_PATH="$PROJECT_ROOT/evaluation_results/eval_results_common_match_comparison_CNCAP_roi1/mono3d/20260211_120939/evaluation_report.json"
MODEL2_PATH="$PROJECT_ROOT/evaluation_results/eval_results_common_match_comparison_CNCAP_roi1/yolov5s-300w-newdata/20260211_120939/evaluation_report.json"
OUTPUT_PATH="$PROJECT_ROOT/evaluation_results/eval_results_common_match_comparison_CNCAP_roi1/per_case_2d_comparison.json"
echo "Comparing per_case_2d metrics..."
echo "Model 1: mono3d"
echo "Model 2: yolov5s-300w-newdata"
echo ""
python "$SCRIPT_DIR/compare_per_case_2d.py" \
--model1 "$MODEL1_PATH" \
--model2 "$MODEL2_PATH" \
--model1-name "mono3d" \
--model2-name "yolov5s-300w-newdata" \
--threshold 0.1 \
--output "$OUTPUT_PATH" \
--top-n 30
echo ""
echo "Done! Results saved to: $OUTPUT_PATH"

View File

@@ -0,0 +1,439 @@
#!/usr/bin/env python3
"""
Find Common Matches Between Two Models
This tool finds the common GT objects that were successfully matched by both models,
enabling fair comparison of 3D prediction quality on the same set of targets.
Usage:
python eval_tools/find_common_matches.py \
--model1-matches eval_results/model1/detailed_3d_matches.json \
--model2-matches eval_results/model2/detailed_3d_matches.json \
--output common_matches.json
"""
import argparse
import json
import sys
from pathlib import Path
from collections import defaultdict
import numpy as np
def find_common_matches(model1_matches, model2_matches):
"""
Find common GT objects matched by both models.
Args:
model1_matches: dict, detailed matches from model 1
model2_matches: dict, detailed matches from model 2
Returns:
tuple: (common_matches, stats)
common_matches: dict with structure {case: {frame: {class: [match_info]}}}
stats: dict with match statistics
"""
common_matches = {}
stats = {
'model1_total': 0,
'model2_total': 0,
'common': 0,
'model1_unique': 0,
'model2_unique': 0,
'per_class': {}
}
# Iterate through all cases in model 1
for case_name in model1_matches:
if case_name not in model2_matches:
# Case not in model 2, skip
continue
common_matches[case_name] = {}
# Iterate through frames
for frame_name in model1_matches[case_name]:
if frame_name not in model2_matches[case_name]:
# Frame not in model 2, skip
continue
common_matches[case_name][frame_name] = {}
# Iterate through classes
for class_name in model1_matches[case_name][frame_name]:
if class_name not in model2_matches[case_name][frame_name]:
# Class not in model 2, skip
continue
# Get match lists for this class
m1_list = model1_matches[case_name][frame_name][class_name]
m2_list = model2_matches[case_name][frame_name][class_name]
# Build GT ID to index mappings
m1_gt_ids = {m['gt_id']: i for i, m in enumerate(m1_list)}
m2_gt_ids = {m['gt_id']: i for i, m in enumerate(m2_list)}
# Find common GT IDs
common_gt_ids = set(m1_gt_ids.keys()) & set(m2_gt_ids.keys())
# Update statistics
if class_name not in stats['per_class']:
stats['per_class'][class_name] = {
'model1_total': 0,
'model2_total': 0,
'common': 0,
'model1_unique': 0,
'model2_unique': 0
}
stats['model1_total'] += len(m1_list)
stats['model2_total'] += len(m2_list)
stats['common'] += len(common_gt_ids)
stats['model1_unique'] += len(m1_gt_ids) - len(common_gt_ids)
stats['model2_unique'] += len(m2_gt_ids) - len(common_gt_ids)
stats['per_class'][class_name]['model1_total'] += len(m1_list)
stats['per_class'][class_name]['model2_total'] += len(m2_list)
stats['per_class'][class_name]['common'] += len(common_gt_ids)
stats['per_class'][class_name]['model1_unique'] += len(m1_gt_ids) - len(common_gt_ids)
stats['per_class'][class_name]['model2_unique'] += len(m2_gt_ids) - len(common_gt_ids)
# Store common match information
common_list = []
for gt_id in common_gt_ids:
common_list.append({
'gt_id': gt_id,
'model1_idx': m1_gt_ids[gt_id],
'model2_idx': m2_gt_ids[gt_id]
})
if common_list:
common_matches[case_name][frame_name][class_name] = common_list
# Calculate percentages
if stats['model1_total'] > 0:
stats['common_percentage_of_model1'] = (stats['common'] / stats['model1_total']) * 100
else:
stats['common_percentage_of_model1'] = 0
if stats['model2_total'] > 0:
stats['common_percentage_of_model2'] = (stats['common'] / stats['model2_total']) * 100
else:
stats['common_percentage_of_model2'] = 0
for class_name in stats['per_class']:
class_stats = stats['per_class'][class_name]
if class_stats['model1_total'] > 0:
class_stats['common_percentage_of_model1'] = (class_stats['common'] / class_stats['model1_total']) * 100
else:
class_stats['common_percentage_of_model1'] = 0
if class_stats['model2_total'] > 0:
class_stats['common_percentage_of_model2'] = (class_stats['common'] / class_stats['model2_total']) * 100
else:
class_stats['common_percentage_of_model2'] = 0
return common_matches, stats
# Default distance ranges matching eval metrics_3d config
DEFAULT_LONG_RANGES = [
(0, 10), (10, 20), (20, 30), (30, 40), (40, 50),
(50, 60), (60, 70), (70, 80), (80, 90), (90, 100), (100, 999)
]
DEFAULT_LAT_RANGES = [
(-50, -40), (-40, -30), (-30, -20), (-20, -10), (-10, 0),
(0, 10), (10, 20), (20, 30), (30, 40), (40, 50)
]
def _range_key_long(lo, hi):
return f'long_{lo}-{hi}m'
def _range_key_lat(lo, hi):
return f'lat_{lo}-{hi}m'
def _make_stats(data_dict):
"""Compute mean/std/median/min/max for each list in data_dict."""
result = {}
for key, values in data_dict.items():
if key in ('samples',):
result[key] = values
elif isinstance(values, list) and len(values) > 0:
arr = np.array(values)
result[key] = {
'mean': float(np.mean(arr)),
'std': float(np.std(arr)),
'median': float(np.median(arr)),
'min': float(np.min(arr)),
'max': float(np.max(arr)),
}
return result
def _empty_bucket():
return {
'lateral': [], 'longitudinal': [], 'longitudinal_relative': [],
'heading': [], 'heading_relaxed': [], 'is_reversal': [], 'samples': 0
}
def _finalize_class_stats(data):
"""Convert a bucket dict to stats dict, adding optional fields."""
entry = {
'num_samples': data['samples'],
'lateral_error': _make_stats({'lateral': data['lateral']})['lateral'],
'longitudinal_error': _make_stats({'longitudinal': data['longitudinal']})['longitudinal'],
'heading_error': _make_stats({'heading': data['heading']})['heading'],
}
if data['longitudinal_relative']:
entry['longitudinal_relative_error'] = _make_stats(
{'v': data['longitudinal_relative']})['v']
if data['heading_relaxed']:
entry['heading_error_relaxed'] = _make_stats(
{'v': data['heading_relaxed']})['v']
if data['is_reversal']:
count = int(sum(data['is_reversal']))
entry['reversal_count'] = count
entry['reversal_percentage'] = float(count / data['samples'] * 100) if data['samples'] > 0 else 0.0
return entry
def recompute_3d_stats_from_common_matches(matches_data, common_matches, model_name,
long_ranges=None, lat_ranges=None):
"""
Recompute 3D statistics based on common matches only.
Returns a dict with structure:
{
class_name: {
'overall': { ... },
'long_0-10m': { ... },
...
'lat_-10-0m': { ... },
...
}
}
"""
if long_ranges is None:
long_ranges = DEFAULT_LONG_RANGES
if lat_ranges is None:
lat_ranges = DEFAULT_LAT_RANGES
# Bucket structure: class -> range_key -> _empty_bucket()
overall = {} # class -> _empty_bucket()
by_long = {} # class -> range_key -> _empty_bucket()
by_lat = {} # class -> range_key -> _empty_bucket()
for case_name, frames in common_matches.items():
for frame_name, classes in frames.items():
for class_name, common_list in classes.items():
if class_name not in overall:
overall[class_name] = _empty_bucket()
by_long[class_name] = {
_range_key_long(lo, hi): _empty_bucket()
for lo, hi in long_ranges
}
by_lat[class_name] = {
_range_key_lat(lo, hi): _empty_bucket()
for lo, hi in lat_ranges
}
for match_info in common_list:
idx = match_info[f'{model_name}_idx']
match = matches_data[case_name][frame_name][class_name][idx]
errs = match['errors']
dist = match.get('distance', {})
z_val = dist.get('longitudinal', None)
x_val = dist.get('lateral', None)
# Helper: fill one bucket
def _fill(bucket):
bucket['lateral'].append(errs['lateral'])
bucket['longitudinal'].append(errs['longitudinal'])
bucket['heading'].append(errs['heading'])
if 'longitudinal_relative' in errs:
bucket['longitudinal_relative'].append(errs['longitudinal_relative'])
if 'heading_relaxed' in errs:
bucket['heading_relaxed'].append(errs['heading_relaxed'])
if 'is_reversal' in errs:
bucket['is_reversal'].append(errs['is_reversal'])
bucket['samples'] += 1
_fill(overall[class_name])
# Longitudinal range bucket
if z_val is not None:
for lo, hi in long_ranges:
if lo <= z_val < hi:
_fill(by_long[class_name][_range_key_long(lo, hi)])
break
# Lateral range bucket
if x_val is not None:
for lo, hi in lat_ranges:
if lo <= x_val < hi:
_fill(by_lat[class_name][_range_key_lat(lo, hi)])
break
# Build result
result = {}
for class_name in overall:
result[class_name] = {}
# overall
if overall[class_name]['samples'] > 0:
result[class_name]['overall'] = _finalize_class_stats(overall[class_name])
else:
result[class_name]['overall'] = {'num_samples': 0}
# per longitudinal range
for rk, bucket in by_long[class_name].items():
if bucket['samples'] > 0:
result[class_name][rk] = _finalize_class_stats(bucket)
# per lateral range
for rk, bucket in by_lat[class_name].items():
if bucket['samples'] > 0:
result[class_name][rk] = _finalize_class_stats(bucket)
return result
def print_statistics(stats, model1_name='model1', model2_name='model2'):
"""Print match statistics in a readable format."""
print("\n" + "="*80)
print("COMMON MATCH STATISTICS")
print("="*80)
print(f"\nOverall:")
print(f" {model1_name} Total Matches: {stats['model1_total']:,}")
print(f" {model2_name} Total Matches: {stats['model2_total']:,}")
print(f" Common Matches: {stats['common']:,} ({stats['common_percentage_of_model1']:.1f}% of {model1_name})")
print(f" {model1_name} Unique: {stats['model1_unique']:,} ({100 - stats['common_percentage_of_model1']:.1f}%)")
print(f" {model2_name} Unique: {stats['model2_unique']:,} ({100 - stats['common_percentage_of_model2']:.1f}%)")
print(f"\nPer-Class Statistics:")
# Truncate model names if too long for column headers
m1_short = model1_name[:10]
m2_short = model2_name[:10]
print(f"{'Class':<15} {m1_short:>10} {m2_short:>10} {'Common':>10} {'Common%':>10} {m1_short+' Uniq':>12} {m2_short+' Uniq':>12}")
print("-" * 80)
for class_name, class_stats in sorted(stats['per_class'].items()):
print(f"{class_name:<15} {class_stats['model1_total']:>10,} {class_stats['model2_total']:>10,} "
f"{class_stats['common']:>10,} {class_stats['common_percentage_of_model1']:>9.1f}% "
f"{class_stats['model1_unique']:>12,} {class_stats['model2_unique']:>12,}")
def main():
"""Main function."""
parser = argparse.ArgumentParser(
description='Find common matches between two model evaluation results',
formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument('--model1-matches', type=str, required=True,
help='Path to model 1 detailed_3d_matches.json file')
parser.add_argument('--model2-matches', type=str, required=True,
help='Path to model 2 detailed_3d_matches.json file')
parser.add_argument('--output', type=str, default='common_matches.json',
help='Output path for common matches JSON file')
parser.add_argument('--model1-name', type=str, default='model1',
help='Name for model 1 (default: model1)')
parser.add_argument('--model2-name', type=str, default='model2',
help='Name for model 2 (default: model2)')
args = parser.parse_args()
# Load detailed matches
print(f"Loading model 1 matches from: {args.model1_matches}")
with open(args.model1_matches, 'r') as f:
model1_matches = json.load(f)
print(f"Loading model 2 matches from: {args.model2_matches}")
with open(args.model2_matches, 'r') as f:
model2_matches = json.load(f)
# Find common matches
print("\nFinding common matches...")
common_matches, stats = find_common_matches(model1_matches, model2_matches)
# Print statistics
print_statistics(stats, args.model1_name, args.model2_name)
# Recompute 3D stats for common matches
print("\nRecomputing 3D statistics for common matches...")
model1_stats = recompute_3d_stats_from_common_matches(model1_matches, common_matches, 'model1')
model2_stats = recompute_3d_stats_from_common_matches(model2_matches, common_matches, 'model2')
# Prepare output
output_data = {
'match_statistics': stats,
'common_matches': common_matches,
'model1_3d_stats': model1_stats,
'model2_3d_stats': model2_stats,
'model_names': {
'model1': args.model1_name,
'model2': args.model2_name
}
}
# Save output
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, 'w') as f:
json.dump(output_data, f, indent=2)
print(f"\n✓ Common matches saved to: {output_path}")
# Print 3D stats comparison
print("\n" + "="*80)
print("3D STATISTICS COMPARISON (COMMON MATCHES ONLY)")
print("="*80)
for class_name in sorted(model1_stats.keys()):
if class_name not in model2_stats:
continue
# New format: class stats are nested under 'overall'
m1 = model1_stats[class_name].get('overall', model1_stats[class_name])
m2 = model2_stats[class_name].get('overall', model2_stats[class_name])
print(f"\n{class_name.upper()} (n={m1.get('num_samples', 0):,}):")
print(f"{'Metric':<20} {args.model1_name:>15} {args.model2_name:>15} {'Diff':>12} {'Change %':>10}")
print("-" * 80)
for error_type in ['lateral_error', 'longitudinal_error', 'heading_error']:
if error_type not in m1 or error_type not in m2:
continue
m1_mean = m1[error_type]['mean']
m2_mean = m2[error_type]['mean']
diff = m2_mean - m1_mean
change_pct = (diff / m1_mean * 100) if m1_mean > 0 else 0
error_name = error_type.replace('_', ' ').title()
print(f"{error_name:<20} {m1_mean:>15.4f} {m2_mean:>15.4f} {diff:>+12.4f} {change_pct:>+9.2f}%")
# Print relaxed heading error if available
if 'heading_error_relaxed' in m1 and 'heading_error_relaxed' in m2:
m1_mean = m1['heading_error_relaxed']['mean']
m2_mean = m2['heading_error_relaxed']['mean']
diff = m2_mean - m1_mean
change_pct = (diff / m1_mean * 100) if m1_mean > 0 else 0
print(f"{'Heading Error (Rlx)':<20} {m1_mean:>15.4f} {m2_mean:>15.4f} {diff:>+12.4f} {change_pct:>+9.2f}%")
# Print reversal statistics if available
if 'reversal_count' in m1 and 'reversal_count' in m2:
m1_count = m1['reversal_count']
m1_pct = m1.get('reversal_percentage', 0)
m2_count = m2['reversal_count']
m2_pct = m2.get('reversal_percentage', 0)
print(f"{'Reversals':<20} {m1_count:>11} ({m1_pct:>5.1f}%) {m2_count:>11} ({m2_pct:>5.1f}%)")
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,34 @@
#!/bin/bash
#
# 根据 evaluation_report.json 自动生成单模型 Markdown 评测报告
#
# 用法:
# bash eval_tools/model_comparison/gen_eval_report.sh <evaluation_report.json>
# bash eval_tools/model_comparison/gen_eval_report.sh <evaluation_report.json> [--output <输出路径>] [--model "模型名"] [--date YYYY-MM-DD]
#
# 示例:
# bash eval_tools/model_comparison/gen_eval_report.sh \
# evaluation_results/.../evaluation_report.json
#
# bash eval_tools/model_comparison/gen_eval_report.sh \
# evaluation_results/.../evaluation_report.json \
# --model yolov5s-300w-newdata-cncap \
# --date 2026-02-28
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
if [ $# -eq 0 ]; then
echo "用法: bash $0 <evaluation_report.json> [选项...]"
echo ""
echo "选项:"
echo " --output/-o <路径> 输出 Markdown 文件路径默认JSON 同目录下的 EVALUATION_REPORT.md"
echo " --model <名称> 模型名称(默认从目录名推断)"
echo " --date <YYYY-MM-DD> 评测日期(默认:今天)"
exit 1
fi
python "${SCRIPT_DIR}/generate_eval_report.py" "$@"

View File

@@ -0,0 +1,35 @@
#!/bin/bash
#
# 根据 comparison_report.json 自动生成 Markdown 评测报告
#
# 用法:
# bash eval_tools/model_comparison/gen_report.sh <comparison_report.json>
# bash eval_tools/model_comparison/gen_report.sh <comparison_report.json> [--output <输出路径>] [--title "标题"] [--background "背景说明"] [--date YYYY-MM-DD]
#
# 示例:
# bash eval_tools/model_comparison/gen_report.sh \
# evaluation_results/eval_results_.../comparison_common_matches_.../comparison_report.json
#
# bash eval_tools/model_comparison/gen_report.sh \
# evaluation_results/.../comparison_report.json \
# --output my_report.md \
# --title "ROI0 模型对比报告"
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH}"
if [ $# -eq 0 ]; then
echo "用法: bash $0 <comparison_report.json> [选项...]"
echo ""
echo "选项:"
echo " --output/-o <路径> 输出 Markdown 文件路径默认JSON 同目录下的 COMPARISON_REPORT.md"
echo " --title <标题> 自定义报告标题"
echo " --background <说明> 背景说明文字"
echo " --date <YYYY-MM-DD> 评测日期(默认:今天)"
exit 1
fi
python "${SCRIPT_DIR}/generate_comparison_report.py" "$@"

View File

@@ -0,0 +1,496 @@
#!/usr/bin/env python3
"""
自动将 comparison_report.json 转换为中文 Markdown 评测报告。
用法:
python generate_comparison_report.py <comparison_report.json 路径>
python generate_comparison_report.py <comparison_report.json 路径> --output <输出文件路径>
python generate_comparison_report.py <comparison_report.json 路径> --title "自定义标题"
python generate_comparison_report.py <comparison_report.json 路径> --background "背景说明文字"
python generate_comparison_report.py <comparison_report.json 路径> --date 2026-03-01
示例:
python generate_comparison_report.py \
evaluation_results/eval_results_common_match_comparison_cncap_yolov5s_20260228_roi0/comparison_common_matches_20260228_102849/comparison_report.json
python generate_comparison_report.py \
evaluation_results/.../comparison_report.json \
--output my_report.md \
--title "ROI1 模型对比报告"
"""
import json
import re
import argparse
import sys
from datetime import date
from pathlib import Path
# Allow importing class_config from the eval_tools root
sys.path.insert(0, str(Path(__file__).parent.parent))
from class_config import REPORT_3D_CLASS_LABELS
# ── 阈值设置 ─────────────────────────────────────────────────────────────────
# AP 差异超过此阈值才标记为"优",否则标记为"持平"
AP_TIE_THRESHOLD = 0.005 # 0.5%
METRIC_TIE_THRESHOLD = 0.005 # 用于 precision/recall/f1 的判断阈值(绝对值)
ERROR_TIE_THRESHOLD_REL = 2.0 # 3D 误差相对变化(%)小于此值视为持平
def fmt(v: float, decimals: int = 4) -> str:
return f"{v:.{decimals}f}"
def fmt_pct(v: float) -> str:
sign = "+" if v >= 0 else ""
return f"{sign}{v:.2f}%"
def fmt_diff(v: float) -> str:
sign = "+" if v >= 0 else ""
return f"{sign}{v:.4f}"
def judge(diff: float, rel: float, higher_is_better: bool = True,
abs_thr: float = AP_TIE_THRESHOLD, rel_thr: float = None,
model1_name: str = "model1", model2_name: str = "model2") -> str:
"""
根据 diff (model2 - model1) 判断哪个模型更好。
higher_is_better=True → diff>0 代表 model2 更好
higher_is_better=False → diff<0 代表 model2 更好(即误差更小)
"""
if rel_thr is not None:
tie = abs(rel) < rel_thr
else:
tie = abs(diff) < abs_thr
if tie:
return "⚖️ 持平"
m2_better = (diff > 0) if higher_is_better else (diff < 0)
m2_short = model2_name.split("-")[-1] # e.g. "cncap"
m1_short = model1_name.split("-")[-1] # e.g. "newdata", "mono3d"
if m2_better:
return f"{m2_short}"
else:
return f"{m1_short}"
def build_report(data: dict, model1: str, model2: str,
report_date: str, title: str = None, background: str = None) -> str:
"""生成完整 Markdown 报告字符串。"""
m2d = data["2d_metrics"]
m3d = data.get("3d_metrics", {})
stats = data.get("match_statistics", {})
summary = data.get("summary", {})
# ── 名称简写 ──────────────────────────────────────────────────────────────
m1_short = model1
m2_short = model2
# 取最后一段作为简称用于表格
m1_tag = model1.split("-")[-1] # e.g. "newdata"
m2_tag = model2.split("-")[-1] # e.g. "cncap"
lines = []
# ── 标题 ─────────────────────────────────────────────────────────────────
auto_title = title or f"模型对比Overall指标总结 ({model1} vs {model2} - 通用数据集评测)"
lines.append(f"# {auto_title}")
lines.append("")
lines.append(f"**对比模型**: {model1} vs {model2} ")
lines.append(f"**评测日期**: {report_date} ")
lines.append(f"**数据集**: 通用数据集 (Common Match Cases) ")
total_common = stats.get("common", None)
m1_total = stats.get("model1_total", None)
m2_total = stats.get("model2_total", None)
if total_common is not None and m1_total is not None and m2_total is not None:
lines.append(f"**匹配样本**: {total_common:,} ({model1}: {m1_total:,} | {model2}: {m2_total:,})")
else:
lines.append(f"**匹配样本**: N/A")
lines.append("")
if background:
lines.append(f"> **背景说明**: {background}")
else:
lines.append(f"> **背景说明**: 本次评测对比了 {model1}{model2}评估两者在通用数据集上的2D/3D检测性能差异。")
lines.append("")
lines.append("---")
lines.append("")
# ── 2D Overall ────────────────────────────────────────────────────────────
ov = m2d["overall"]
lines.append("## 📊 2D检测指标 (Overall)")
lines.append("")
lines.append("### 总体性能对比")
lines.append("")
lines.append(f"| 指标 | {model1} | {model2} | 差异 | 相对变化 | 结果 |")
lines.append("|------|" + "---|" * 5)
def ov_row(key, label, higher_is_better=True):
v1 = ov[key][model1]
v2 = ov[key][model2]
diff = ov[key]["diff"]
rel = ov[key]["relative_change_%"]
j = judge(diff, rel, higher_is_better, abs_thr=METRIC_TIE_THRESHOLD,
model1_name=model1, model2_name=model2)
return f"| **{label}** | {fmt(v1)} | {fmt(v2)} | {fmt_diff(diff)} | {fmt_pct(rel)} | {j} |"
lines.append(ov_row("precision", "PRECISION"))
lines.append(ov_row("recall", "RECALL"))
lines.append(ov_row("f1_score", "F1-Score"))
lines.append(ov_row("map", "mAP"))
lines.append("")
# 关键发现
prec_diff = ov["precision"]["relative_change_%"]
rec_diff = ov["recall"]["relative_change_%"]
map_diff = ov["map"]["relative_change_%"]
f1_diff = ov["f1_score"]["relative_change_%"]
ap_wins = summary.get("2d", {}).get("ap", {}).get("wins", "?")
ap_losses = summary.get("2d", {}).get("ap", {}).get("losses", "?")
ap_ties = summary.get("2d", {}).get("ap", {}).get("ties", "?")
lines.append("### 关键发现")
lines.append("")
lines.append(f"- 📊 **Precision**: {model2}{'领先' if prec_diff > 0 else '落后'}{fmt_pct(abs(prec_diff))}{'误检率略低' if prec_diff > 0 else '误检率略高'}")
lines.append(f"- 📊 **Recall**: {model1 if rec_diff < 0 else model2}领先{fmt_pct(abs(rec_diff))},检出率{'更高' if rec_diff < 0 else '更低'}")
lines.append(f"- 📊 **mAP**: {model2 if map_diff > 0 else model1}领先{fmt_pct(abs(map_diff))}{'极小差异,基本持平' if abs(map_diff) < 2 else '有一定差距'}")
lines.append(f"- 📊 **F1-Score**: {'两模型基本持平' if abs(f1_diff) < 1 else (model2 + '更优' if f1_diff > 0 else model1 + '更优')}(差距{fmt_pct(abs(f1_diff))}")
lines.append(f"- ⚖️ **类别赢负统计 (AP)**: {m2_tag}{ap_wins}类, {m1_tag}{ap_losses}类, 平局{ap_ties}")
lines.append("")
lines.append("---")
lines.append("")
# ── 2D Per-Class ──────────────────────────────────────────────────────────
pc = m2d.get("per_class", {})
lines.append("## 📋 2D检测指标 (Per Class)")
lines.append("")
lines.append("### 各类别性能对比")
lines.append("")
lines.append(f"| 类别 | Precision ({m1_tag}) | Precision ({m2_tag}) | Recall ({m1_tag}) | Recall ({m2_tag}) | F1 ({m1_tag}) | F1 ({m2_tag}) | AP ({m1_tag}) | AP ({m2_tag}) | AP差异 | 结果 |")
lines.append("|------|" + "---|" * 10)
adv_m2 = [] # model2 明显更好的类别
adv_m1 = [] # model1 明显更好的类别
for cls, cd in pc.items():
prec1 = cd["precision"][model1]
prec2 = cd["precision"][model2]
rec1 = cd["recall"][model1]
rec2 = cd["recall"][model2]
f1_1 = cd["f1_score"][model1]
f1_2 = cd["f1_score"][model2]
ap1 = cd["ap"][model1]
ap2 = cd["ap"][model2]
ap_d = cd["ap"]["diff"]
ap_r = cd["ap"]["relative_change_%"]
j = judge(ap_d, ap_r, True, abs_thr=AP_TIE_THRESHOLD,
model1_name=model1, model2_name=model2)
lines.append(
f"| **{cls}** | {fmt(prec1)} | {fmt(prec2)} | {fmt(rec1)} | {fmt(rec2)} "
f"| {fmt(f1_1)} | {fmt(f1_2)} | {fmt(ap1)} | {fmt(ap2)} | {fmt_diff(ap_d)} | {j} |"
)
if abs(ap_r) >= 2.0: # 相对变化>=2%才算显著
if ap_d > 0:
adv_m2.append((cls, ap1, ap2, ap_r))
elif ap_d < 0:
adv_m1.append((cls, ap1, ap2, ap_r))
lines.append("")
lines.append("### 类别分析")
lines.append("")
if adv_m2:
lines.append(f"**{model2} 优势类别** (AP更高):")
for cls, ap1, ap2, rel in sorted(adv_m2, key=lambda x: -x[3]):
mark = "**大幅领先**" if rel > 8 else "领先"
lines.append(f"- {cls}: {m2_tag} {fmt(ap2)} > {m1_tag} {fmt(ap1)}{mark}{fmt_pct(rel)}")
lines.append("")
if adv_m1:
lines.append(f"**{model1} 优势类别** (AP更高):")
for cls, ap1, ap2, rel in sorted(adv_m1, key=lambda x: x[3]):
mark = "**大幅领先**" if abs(rel) > 8 else "领先"
lines.append(f"- {cls}: {m1_tag} {fmt(ap1)} > {m2_tag} {fmt(ap2)}{mark}{fmt_pct(abs(rel))}")
lines.append("")
lines.append("---")
lines.append("")
# ── 3D Metrics ────────────────────────────────────────────────────────────
if m3d:
lines.append("## 🎯 3D检测指标")
lines.append("")
cls_labels = REPORT_3D_CLASS_LABELS
for cls_key, cls_label in cls_labels.items():
if cls_key not in m3d:
continue
cd = m3d[cls_key]
ov3 = cd.get("overall", {})
if not ov3:
continue
n = cd.get("common_samples")
n_str = f"{n:,} 个样本" if n is not None else "N/A 个样本"
lines.append(f"### {cls_label} - {n_str}")
lines.append("")
lines.append(f"| 指标 | {model1} | {model2} | 差异 | 相对变化 | 结果 |")
lines.append("|------|" + "---|" * 5)
def row3d(key, label, higher_is_better=False):
if key not in ov3:
return None
v1 = ov3[key][model1]["mean"]
v2 = ov3[key][model2]["mean"]
diff = ov3[key]["diff"]
rel = ov3[key]["relative_change_%"]
j = judge(diff, rel, higher_is_better,
rel_thr=ERROR_TIE_THRESHOLD_REL,
model1_name=model1, model2_name=model2)
return f"| **{label}** | {fmt(v1)} | {fmt(v2)} | {fmt_diff(diff)} | {fmt_pct(rel)} | {j} |"
for row in [
row3d("lateral_error", "Lateral Error"),
row3d("longitudinal_error", "Longitudinal Error"),
row3d("longitudinal_relative_error", "Longitudinal Relative Error"),
row3d("heading_error", "Heading Error"),
row3d("heading_error_relaxed", "Heading Error Relaxed"),
]:
if row is not None:
lines.append(row)
if "reversal_info" in ov3:
rev1 = ov3["reversal_info"][model1]
rev2 = ov3["reversal_info"][model2]
rev_j = "" + (m2_tag if rev2["percentage"] < rev1["percentage"] else m1_tag) + ""
if abs(rev1["percentage"] - rev2["percentage"]) < 0.5:
rev_j = "⚖️ 持平"
lines.append(
f"| **Reversal Cases** | {rev1['count']:,} ({rev1['percentage']:.2f}%) "
f"| {rev2['count']:,} ({rev2['percentage']:.2f}%) | - | - | {rev_j} |"
)
lines.append("")
# ── 纵向区间对比 ──────────────────────────────────────────────
def _long_sort_key(k):
stripped = k[len("long_"):].replace("m", "")
m = re.search(r'(?<=\d)-', stripped)
if m:
try:
return float(stripped[:m.start()])
except ValueError:
pass
return float('inf')
long_keys = sorted(
[k for k in cd.keys() if k.startswith("long_")],
key=_long_sort_key
)
if long_keys:
lines.append(f"#### 纵向区间对比")
lines.append("")
lines.append(
f"| 区间 | 样本数 "
f"| Lat ({m1_tag}) | Lat ({m2_tag}) | Lat Δ% "
f"| Long ({m1_tag}) | Long ({m2_tag}) | Long Δ% "
f"| LongRel ({m1_tag}) | LongRel ({m2_tag}) | LongRel Δ% "
f"| Head ({m1_tag}) | Head ({m2_tag}) | Head Δ% |"
)
lines.append("|------|" + "---|" * 13)
for rk in long_keys:
rb = cd[rk]
if not rb:
continue
def _rv(metric, model):
d = rb.get(metric, {})
if model in d:
return fmt(d[model]["mean"])
return "-"
def _rd(metric):
d = rb.get(metric, {})
rel = d.get("relative_change_%")
if rel is None:
return "-"
return fmt_pct(rel)
# sample count from any available metric
n_range = "-"
for _mk in ("lateral_error", "longitudinal_error", "heading_error"):
_md = rb.get(_mk, {})
if model1 in _md and "samples" in _md[model1]:
n_range = f"{_md[model1]['samples']:,}"
break
# range label: strip prefix and trailing 'm'
rl = rk[len("long_"):]
lines.append(
f"| **{rl}** | {n_range} "
f"| {_rv('lateral_error', model1)} | {_rv('lateral_error', model2)} | {_rd('lateral_error')} "
f"| {_rv('longitudinal_error', model1)} | {_rv('longitudinal_error', model2)} | {_rd('longitudinal_error')} "
f"| {_rv('longitudinal_relative_error', model1)} | {_rv('longitudinal_relative_error', model2)} | {_rd('longitudinal_relative_error')} "
f"| {_rv('heading_error', model1)} | {_rv('heading_error', model2)} | {_rd('heading_error')} |"
)
lines.append("")
lines.append("---")
lines.append("")
# ── Match Statistics ──────────────────────────────────────────────────────
if stats:
lines.append("## 📊 样本匹配统计")
lines.append("")
lines.append("### 整体匹配情况")
lines.append("")
lines.append("| 模型 | 总样本数 | 公共样本 | 独有样本 | 公共占比 |")
lines.append("|------|----------|----------|----------|---------|")
m1_pct = stats.get("common_percentage_of_model1", 0)
m2_pct = stats.get("common_percentage_of_model2", 0)
m1_uniq = stats.get("model1_unique", 0)
m2_uniq = stats.get("model2_unique", 0)
lines.append(f"| **{model1}** | {m1_total:,} | {total_common:,} | {m1_uniq:,} | {m1_pct:.2f}% |")
lines.append(f"| **{model2}** | {m2_total:,} | {total_common:,} | {m2_uniq:,} | {m2_pct:.2f}% |")
lines.append("")
per_cls_stats = stats.get("per_class", {})
if per_cls_stats:
lines.append("### 各类别匹配情况 (3D)")
lines.append("")
lines.append(f"| 类别 | {m1_tag}总数 | {m2_tag}总数 | 公共样本 | {m1_tag}占比 | {m2_tag}占比 |")
lines.append("|------|" + "---|" * 5)
for cls, cs in per_cls_stats.items():
lines.append(
f"| **{cls}** | {cs['model1_total']:,} | {cs['model2_total']:,} "
f"| {cs['common']:,} | {cs['common_percentage_of_model1']:.2f}% "
f"| {cs['common_percentage_of_model2']:.2f}% |"
)
lines.append("")
lines.append("---")
lines.append("")
# ── Summary / Conclusions ─────────────────────────────────────────────────
lines.append("## 🎯 结论与建议")
lines.append("")
lines.append("### 2D检测汇总")
lines.append("")
sum2d = summary.get("2d", {})
ap_w = sum2d.get("ap", {}).get("wins", 0)
ap_l = sum2d.get("ap", {}).get("losses", 0)
ap_t = sum2d.get("ap", {}).get("ties", 0)
f1_w = sum2d.get("f1_score", {}).get("wins", 0)
f1_l = sum2d.get("f1_score", {}).get("losses", 0)
f1_t = sum2d.get("f1_score", {}).get("ties", 0)
lines.append(f"- **AP 类别统计**: {m2_tag}{ap_w}类 / {m1_tag}{ap_l}类 / 平局{ap_t}")
lines.append(f"- **F1 类别统计**: {m2_tag}{f1_w}类 / {m1_tag}{f1_l}类 / 平局{f1_t}")
lines.append(f"- **整体mAP**: {model1}={fmt(ov['map'][model1])} vs {model2}={fmt(ov['map'][model2])} ({fmt_pct(ov['map']['relative_change_%'])})")
lines.append("")
if m3d:
sum3d = summary.get("3d", {})
lat_w = sum3d.get("lateral", {}).get("wins", 0)
lat_l = sum3d.get("lateral", {}).get("losses", 0)
lon_w = sum3d.get("longitudinal", {}).get("wins", 0)
lon_l = sum3d.get("longitudinal", {}).get("losses", 0)
hd_w = sum3d.get("heading", {}).get("wins", 0)
hd_l = sum3d.get("heading", {}).get("losses", 0)
lines.append("### 3D检测汇总")
lines.append("")
lines.append(f"- **横向误差 (Lateral)**: {m2_tag}{lat_w}类 / {m1_tag}{lat_l}")
lines.append(f"- **纵向误差 (Longitudinal)**: {m2_tag}{lon_w}类 / {m1_tag}{lon_l}")
lines.append(f"- **航向误差 (Heading)**: {m2_tag}{hd_w}类 / {m1_tag}{hd_l}")
lines.append("")
lines.append("### 综合建议")
lines.append("")
# 自动判断整体赢家
map_rel = ov["map"]["relative_change_%"]
if map_rel > 2:
overall_winner = model2
elif map_rel < -2:
overall_winner = model1
else:
overall_winner = None
if overall_winner:
lines.append(f"- 🏆 **综合mAP**: {overall_winner} 整体占优({fmt_pct(abs(map_rel))}")
else:
lines.append(f"- ⚖️ **综合mAP**: 两模型基本持平(差距{fmt_pct(abs(map_rel))}")
adv_summary_m2 = [(c, r) for c, *_, r in adv_m2]
adv_summary_m1 = [(c, r) for c, *_, r in adv_m1]
if adv_summary_m2:
cls_str = "".join(c for c, _ in adv_summary_m2)
lines.append(f"- ✅ **{model2} 改善**: {cls_str} 类别AP有所提升")
if adv_summary_m1:
cls_str = "".join(c for c, _ in adv_summary_m1)
lines.append(f"- ⚠️ **{model2} 退化**: {cls_str} 类别AP有所下降")
lines.append("")
return "\n".join(lines)
def main():
parser = argparse.ArgumentParser(
description="将 comparison_report.json 转换为中文 Markdown 评测报告"
)
parser.add_argument("json_path", help="comparison_report.json 的路径")
parser.add_argument("--output", "-o", default=None,
help="输出 Markdown 文件路径(默认与 JSON 同目录,文件名 COMPARISON_REPORT.md")
parser.add_argument("--title", default=None,
help="自定义报告标题")
parser.add_argument("--background", default=None,
help="背景说明文字")
parser.add_argument("--date", default=str(date.today()),
help="评测日期 (默认今天,格式 YYYY-MM-DD)")
args = parser.parse_args()
json_path = Path(args.json_path)
if not json_path.exists():
print(f"错误: 文件不存在: {json_path}", file=sys.stderr)
sys.exit(1)
with open(json_path, "r", encoding="utf-8") as f:
data = json.load(f)
# 自动从 JSON 中读取模型名称
models = list(data["2d_metrics"]["overall"]["precision"].keys())
# 过滤掉 diff / relative_change_% 等非模型 key
skip = {"diff", "relative_change_%"}
models = [m for m in models if m not in skip]
if len(models) < 2:
print("错误: 无法从 JSON 中自动识别模型名称,请检查文件格式。", file=sys.stderr)
sys.exit(1)
model1, model2 = models[0], models[1]
print(f"模型1: {model1}")
print(f"模型2: {model2}")
report = build_report(data, model1, model2,
report_date=args.date,
title=args.title,
background=args.background)
# 输出路径
if args.output:
out_path = Path(args.output)
else:
out_path = json_path.parent / "COMPARISON_REPORT.md"
out_path.write_text(report, encoding="utf-8")
print(f"报告已生成: {out_path}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""
自动将单模型 evaluation_report.json 转换为中文 Markdown 评测报告。
用法:
python generate_eval_report.py <evaluation_report.json 路径>
python generate_eval_report.py <evaluation_report.json 路径> --output <输出路径>
python generate_eval_report.py <evaluation_report.json 路径> --model "模型名称" --date 2026-03-01
示例:
python eval_tools/model_comparison/generate_eval_report.py \
evaluation_results/.../evaluation_report.json \
--model yolov5s-300w-newdata-cncap
"""
import json
import argparse
import sys
from datetime import date
from pathlib import Path
# Allow importing class_config from the eval_tools root
sys.path.insert(0, str(Path(__file__).parent.parent))
from class_config import REPORT_3D_CLASS_LABELS
def fmt(v: float, decimals: int = 4) -> str:
return f"{v:.{decimals}f}"
def fmt2(v: float) -> str:
return f"{v:.2f}"
def pct(v: float) -> str:
return f"{v * 100:.1f}%"
def build_report(data: dict, model_name: str, report_date: str) -> str:
lines = []
# ── 标题 ─────────────────────────────────────────────────────────────────
lines.append(f"# 模型评测报告: {model_name}")
lines.append("")
lines.append(f"**模型**: {model_name} ")
lines.append(f"**评测日期**: {report_date} ")
eval_cfg = data.get("evaluation_config", {})
if eval_cfg:
lines.append(f"**置信度阈值 (P/R/F1)**: {eval_cfg.get('conf_threshold', '-')} ")
lines.append(f"**IoU 阈值**: {eval_cfg.get('iou_threshold', '-')} ")
lines.append(f"**AP 计算方法**: {eval_cfg.get('ap_method', '-')} ")
lines.append("")
lines.append("---")
lines.append("")
# ── 2D Overall ────────────────────────────────────────────────────────────
ov2d = data["2d_evaluation"]["overall"]
lines.append("## 📊 2D检测指标 (Overall)")
lines.append("")
lines.append("| 指标 | 数值 |")
lines.append("|------|------|")
lines.append(f"| **Precision** | {fmt(ov2d['precision'])} |")
lines.append(f"| **Recall** | {fmt(ov2d['recall'])} |")
lines.append(f"| **F1-Score** | {fmt(ov2d['f1_score'])} |")
lines.append(f"| **mAP** | {fmt(ov2d['map'])} |")
lines.append(f"| **TP** | {ov2d['tp']:,} |")
lines.append(f"| **FP** | {ov2d['fp']:,} |")
lines.append(f"| **FN** | {ov2d['fn']:,} |")
lines.append("")
# ── 2D Per-Class ──────────────────────────────────────────────────────────
lines.append("## 📋 2D检测指标 (Per Class)")
lines.append("")
lines.append("| 类别 | Precision | Recall | F1 | AP | GT | TP | FP | FN |")
lines.append("|------|-----------|--------|----|----|-----|-----|-----|-----|")
pc2d = data["2d_evaluation"]["per_class"]
for cls, cd in pc2d.items():
lines.append(
f"| **{cls}** | {fmt(cd['precision'])} | {fmt(cd['recall'])} "
f"| {fmt(cd['f1_score'])} | {fmt(cd['ap'])} "
f"| {cd['num_gt']:,} | {cd['tp']:,} | {cd['fp']:,} | {cd['fn']:,} |"
)
lines.append("")
lines.append("---")
lines.append("")
# ── 3D Overall per class ──────────────────────────────────────────────────
m3d = data.get("3d_evaluation", {})
if not m3d:
return "\n".join(lines)
lines.append("## 🎯 3D检测指标")
lines.append("")
CLS_LABELS = REPORT_3D_CLASS_LABELS
LONG_RANGES = ["long_0-10m", "long_10-20m", "long_20-30m", "long_30-40m",
"long_40-50m", "long_50-60m", "long_60-70m", "long_70-80m",
"long_80-90m", "long_90-100m", "long_100-999m"]
for cls_key, cls_label in CLS_LABELS.items():
if cls_key not in m3d:
continue
cd = m3d[cls_key]
ov = cd["overall"]
n = ov["num_samples"]
lines.append(f"### {cls_label} ({n:,} 样本)")
lines.append("")
# Overall stats
lines.append("#### 总体指标")
lines.append("")
lines.append("| 指标 | Mean | Median | Std | P90 |")
lines.append("|------|------|--------|-----|-----|")
def stat_row(label, key):
s = ov[key]
return (f"| **{label}** | {fmt2(s['mean'])} | {fmt2(s['median'])} "
f"| {fmt2(s['std'])} | {fmt2(s['percentile_90'])} |")
lines.append(stat_row("Lateral Error (m)", "lateral_error"))
lines.append(stat_row("Longitudinal Error (m)", "longitudinal_error"))
lines.append(stat_row("Long. Relative Error", "longitudinal_relative_error"))
lines.append(stat_row("Heading Error (rad)", "heading_error"))
lines.append(stat_row("Heading Error Relaxed", "heading_error_relaxed"))
rev_pct = ov['reversal_percentage']
lines.append(f"| **Reversal** | {ov['reversal_count']:,} ({rev_pct:.2f}%) | - | - | - |")
lines.append("")
# Distance range breakdown
avail_ranges = [r for r in LONG_RANGES if r in cd]
if avail_ranges:
lines.append("#### 按距离分段 (纵向误差 Mean / Lateral Mean / Samples)")
lines.append("")
lines.append("| 距离段 | 样本数 | Lateral (m) | Longitudinal (m) | Long.Rel | Heading | Reversal% |")
lines.append("|--------|--------|-------------|------------------|----------|---------|-----------|")
for rng in avail_ranges:
r = cd[rng]
rov = r
rn = rov["num_samples"]
if rn == 0:
continue
lat = rov["lateral_error"]["mean"]
lon = rov["longitudinal_error"]["mean"]
lrel = rov["longitudinal_relative_error"]["mean"]
hd = rov["heading_error"]["mean"]
rev = rov["reversal_percentage"]
lines.append(
f"| {rng.replace('long_', '')} | {rn:,} | {fmt2(lat)} | {fmt2(lon)} "
f"| {lrel:.3f} | {fmt2(hd)} | {rev:.1f}% |"
)
lines.append("")
lines.append("---")
lines.append("")
return "\n".join(lines)
def main():
parser = argparse.ArgumentParser(
description="将单模型 evaluation_report.json 转换为中文 Markdown 评测报告"
)
parser.add_argument("json_path", help="evaluation_report.json 的路径")
parser.add_argument("--output", "-o", default=None,
help="输出 Markdown 文件路径(默认与 JSON 同目录,文件名 EVALUATION_REPORT.md")
parser.add_argument("--model", default=None,
help="模型名称(默认从目录名推断)")
parser.add_argument("--date", default=str(date.today()),
help="评测日期 (默认今天,格式 YYYY-MM-DD)")
args = parser.parse_args()
json_path = Path(args.json_path).resolve()
if not json_path.exists():
print(f"错误: 文件不存在: {json_path}", file=sys.stderr)
sys.exit(1)
with open(json_path, "r", encoding="utf-8") as f:
data = json.load(f)
# 从路径推断模型名称:取 json 所在目录的上级目录名
if args.model:
model_name = args.model
else:
# e.g. .../yolov5s-300w-newdata-cncap/20260228_102849/evaluation_report.json
model_name = json_path.parent.parent.name
if not model_name or model_name == ".":
model_name = json_path.parent.name
print(f"模型: {model_name}")
report = build_report(data, model_name, args.date)
if args.output:
out_path = Path(args.output)
else:
out_path = json_path.parent / "EVALUATION_REPORT.md"
out_path.write_text(report, encoding="utf-8")
print(f"报告已生成: {out_path}")
if __name__ == "__main__":
main()