yolov26_3d/eval_tools/docs/TWO_LEVEL_PATH_SUPPORT.md

Two-Level Path Support for Model Evaluation

Overview

The evaluation system now supports both 1-level and 2-level directory structures for detection results and ground truth labels. This allows for more flexible organization of test data.

Directory Structures

1-Level Structure (Default)

det_root/
    case1/
        txt_results/
            frame001.txt
            frame002.txt
    case2/
        txt_results/
            frame001.txt

gt_root/
    case1/
        labels/
            frame001.txt
            frame002.txt
    case2/
        labels/
            frame001.txt

2-Level Structure

det_root/
    level1_dir/
        case1/
            txt_results/
                frame001.txt
                frame002.txt
        case2/
            txt_results/
                frame001.txt
    level2_dir/
        case3/
            txt_results/
                frame001.txt

gt_root/
    level1_dir/
        case1/
            labels/
                frame001.txt
                frame002.txt
        case2/
            labels/
                frame001.txt
    level2_dir/
        case3/
            labels/
                frame001.txt

Configuration

YAML Configuration File

Add the path_depth parameter to your config file:

dataset:
  det_path: "/path/to/detection/results"
  gt_path: "/path/to/ground/truth"
  path_depth: 2  # Set to 1 for 1-level, 2 for 2-level structure

Example for 1-level structure:

dataset:
  det_path: "/data1/dongying/Mono3d/G1M3/CNCAP_results/mono3d/evalset_roi0"
  gt_path: "/mnt/mono3d/xdzhu_data/Mono3d/Testdata"
  path_depth: 1  # Default

Example for 2-level structure:

dataset:
  det_path: "/data1/dongying/Mono3d/G1M3/CNCAP_results/mono3d"
  gt_path: "/mnt/mono3d/xdzhu_data/Mono3d/Mono3d_4face_2m_g1m3/driving_png"
  path_depth: 2

Command-Line Arguments

You can also specify the path depth via command line:

# 1-level structure (default)
python eval_tools/core/eval.py \
  --config eval_tools/configs/eval_config_mono3d.yaml

# 2-level structure
python eval_tools/core/eval.py \
  --config eval_tools/configs/eval_config_mono3d.yaml \
  --path-depth 2

# Or without config file
python eval_tools/core/eval.py \
  --det-path /path/to/detections \
  --gt-path /path/to/labels \
  --path-depth 2 \
  --output-dir results

How It Works

1-Level Mode (path_depth=1)

1. Scans det_root for all subdirectories (cases)
2. For each case, looks for txt_results/ subdirectory
3. Matches with corresponding gt_root/case/labels/ directory

2-Level Mode (path_depth=2)

1. Scans det_root for all subdirectories (level1 directories)
2. For each level1 directory, scans for case subdirectories
3. For each case, looks for txt_results/ subdirectory
4. Matches with corresponding gt_root/level1/case/labels/ directory

Important: The level1 directory names must match between detection and ground truth paths.

Model Comparison Script

The comparison script automatically inherits the path_depth setting from the config files:

bash eval_tools/model_comparison/compare_models_with_common_matches.sh

The script will:

1. Read path_depth from eval_config_mono3d.yaml for Model 1
2. Read path_depth from eval_config_yolov5s.yaml for Model 2
3. Evaluate both models with their respective path structures
4. Compare results using common matches

Examples

Example 1: Evaluating with 2-level structure

# Update your config file
cat > eval_tools/configs/eval_config_2level.yaml << EOF
dataset:
  det_path: "/data/results/all_models"
  gt_path: "/data/ground_truth/all_datasets"
  path_depth: 2

image:
  width: 1920
  height: 1080

# ... other settings ...
EOF

# Run evaluation
python eval_tools/core/eval.py --config eval_tools/configs/eval_config_2level.yaml

Example 2: Comparing models with different path structures

# Model 1 config (1-level)
dataset:
  det_path: "/data/model1/results"
  gt_path: "/data/gt"
  path_depth: 1

# Model 2 config (2-level)
dataset:
  det_path: "/data/model2/results"
  gt_path: "/data/gt_organized"
  path_depth: 2

Both models can be compared even with different directory structures.

Backward Compatibility

• If path_depth is not specified, it defaults to 1 (1-level structure)
• All existing config files and scripts continue to work without modification
• The system automatically detects and handles both structures

Troubleshooting

Issue: "GT case directory not found"

Cause: Level1 directory names don't match between detection and ground truth paths.

Solution: Ensure that the intermediate directory names are identical:

det_root/dataset_A/case1/  ← "dataset_A" must match
gt_root/dataset_A/case1/   ← "dataset_A" must match

Issue: "No image pairs found"

Cause: Incorrect path_depth setting.

Solution:

• Check your actual directory structure
• Set path_depth: 1 for root/case/txt_results
• Set path_depth: 2 for root/level1/case/txt_results

Issue: Cases are being skipped

Cause: Missing txt_results/ or labels/ subdirectories.

Solution: Verify that each case directory contains:

• Detection: case/txt_results/*.txt
• Ground truth: case/labels/*.txt

Implementation Details

The changes are implemented in:

• eval_tools/evaluator/evaluator.py: Modified load_data_from_paths() method
• eval_tools/core/eval.py: Added --path-depth argument and config support
• eval_tools/configs/*.yaml: Added path_depth parameter

The implementation maintains full backward compatibility while adding support for 2-level structures.