单目3D初始代码
This commit is contained in:
765
tools/scripts_for_gt/profiling/DATASET_ANALYSIS.md
Executable file
765
tools/scripts_for_gt/profiling/DATASET_ANALYSIS.md
Executable file
@@ -0,0 +1,765 @@
|
||||
# YOLOv5-3D 训练数据集分析报告
|
||||
|
||||
## 1. 数据集概览
|
||||
|
||||
### 基本信息
|
||||
- **数据集划分**: 训练集 (train)
|
||||
- **总图像数**: 2,638,689 张
|
||||
- **总标注对象数**: 127,878,960 个
|
||||
- **平均每张图像对象数**: 48.5 个
|
||||
- **标签完整性**: 100% (无缺失标签)
|
||||
|
||||
### 标注格式分布
|
||||
| 格式维度 | 对象数量 | 占比 |
|
||||
|---------|---------|------|
|
||||
| 6维格式 | 89,994,340 | 70.4% |
|
||||
| 50维格式 | 22,070,252 | 17.3% |
|
||||
| 18维格式 | 15,814,368 | 12.4% |
|
||||
|
||||
---
|
||||
|
||||
## 2. 类别分布与统计
|
||||
|
||||
### 2.1 整体类别分布
|
||||
|
||||
| 类别 | 对象数量 | 占比 | 3D标注数量 | 深度标注率 |
|
||||
|-----|---------|------|-----------|-----------|
|
||||
| vehicle (车辆) | 35,607,807 | 27.8% | 22,070,252 | 62.0% |
|
||||
| bicycle (自行车) | 10,658,936 | 8.3% | 6,368,673 | 59.8% |
|
||||
| pedestrian (行人) | 8,086,704 | 6.3% | 3,955,590 | 48.9% |
|
||||
| rider (骑行者) | 6,653,675 | 5.2% | 5,490,105 | 82.5% |
|
||||
| head (人头) | 14,610,038 | 11.4% | - | 0% |
|
||||
| wheel (车轮) | 12,895,577 | 10.1% | - | 0% |
|
||||
| plate (车牌) | 9,939,105 | 7.8% | - | 0% |
|
||||
| roadblock (路障) | 8,058,184 | 6.3% | - | 0% |
|
||||
| guideboard (指示牌) | 6,457,611 | 5.0% | - | 0% |
|
||||
| tl_border (交通灯边框) | 5,645,436 | 4.4% | - | 0% |
|
||||
| tsr (交通标志) | 4,957,683 | 3.9% | - | 0% |
|
||||
| tl_wick (交通灯灯芯) | 3,611,666 | 2.8% | - | 0% |
|
||||
| tl_num (交通灯数字) | 696,538 | 0.5% | - | 0% |
|
||||
|
||||
**关键发现**:
|
||||
- 数据集包含13个类别,覆盖自动驾驶场景的主要元素
|
||||
- 仅4个类别有3D标注(vehicle, bicycle, pedestrian, rider)
|
||||
- 车辆类别数据量最大,占比接近28%
|
||||
- 辅助类别(head, wheel, plate)占比约30%
|
||||
|
||||
---
|
||||
|
||||
## 3. 目标密度分析
|
||||
|
||||
### 3.1 每张图像对象密度
|
||||
|
||||
| 统计量 | 对象数/图 | 类别数/图 |
|
||||
|-------|----------|----------|
|
||||
| 均值 | 48.5 | 7.9 |
|
||||
| 标准差 | 30.0 | 2.8 |
|
||||
| 最小值 | 1 | 1 |
|
||||
| 5%分位 | 8 | 3 |
|
||||
| 25%分位 | 25 | 6 |
|
||||
| 中位数 | 45 | 8 |
|
||||
| 75%分位 | 66 | 10 |
|
||||
| 95%分位 | 103 | 12 |
|
||||
| 最大值 | 252 | 13 |
|
||||
|
||||
**分析**:
|
||||
- 图像平均包含48.5个对象,显示场景复杂度较高
|
||||
- 中位数45个对象,说明数据分布较为均衡
|
||||
- 最密集图像包含252个对象,对检测算法提出挑战
|
||||
- 平均每张图像包含8个类别,场景多样性好
|
||||
|
||||
---
|
||||
|
||||
## 4. 2D边界框特征分析
|
||||
|
||||
### 4.1 主要交通参与者BBox统计
|
||||
|
||||
#### Vehicle (车辆)
|
||||
| 维度 | 宽度(px) | 高度(px) | 面积(px²) | 长宽比 |
|
||||
|-----|---------|---------|----------|-------|
|
||||
| 均值 | 85.4 | 73.2 | 16,238 | 1.41 |
|
||||
| 中位数 | 46 | 36 | 1,584 | 1.24 |
|
||||
| 5%-95%范围 | 10-318 | 8-295 | 98-90,168 | 0.56-2.83 |
|
||||
|
||||
#### Pedestrian (行人)
|
||||
| 维度 | 宽度(px) | 高度(px) | 面积(px²) | 长宽比 |
|
||||
|-----|---------|---------|----------|-------|
|
||||
| 均值 | 25.0 | 70.1 | 3,434 | 0.44 |
|
||||
| 中位数 | 18 | 45 | 799 | 0.38 |
|
||||
| 5%-95%范围 | 8-64 | 16-216 | 144-12,792 | 0.21-0.80 |
|
||||
|
||||
#### Bicycle (自行车)
|
||||
| 维度 | 宽度(px) | 高度(px) | 面积(px²) | 长宽比 |
|
||||
|-----|---------|---------|----------|-------|
|
||||
| 均值 | 58.4 | 77.3 | 7,135 | 0.82 |
|
||||
| 中位数 | 42 | 57 | 2,408 | 0.79 |
|
||||
| 5%-95%范围 | 16-161 | 20-222 | 357-33,600 | 0.39-1.36 |
|
||||
|
||||
#### Rider (骑行者)
|
||||
| 维度 | 宽度(px) | 高度(px) | 面积(px²) | 长宽比 |
|
||||
|-----|---------|---------|----------|-------|
|
||||
| 均值 | 45.5 | 87.2 | 7,560 | 0.55 |
|
||||
| 中位数 | 30 | 56 | 1,680 | 0.49 |
|
||||
| 5%-95%范围 | 9-135 | 18-276 | 162-36,140 | 0.34-0.94 |
|
||||
|
||||
**关键观察**:
|
||||
- 车辆BBox尺寸变化最大(标准差109.7px),反映远近距离差异
|
||||
- 行人长宽比约0.44,符合竖直站立特征
|
||||
- 自行车长宽比接近1,呈方形分布
|
||||
- 骑行者BBox比单独行人更大,符合"人+车"组合特征
|
||||
|
||||
### 4.2 辅助检测目标BBox统计
|
||||
|
||||
#### Head (人头)
|
||||
- 平均尺寸: 12×15 px
|
||||
- 中位数面积: 96 px²
|
||||
- 长宽比: 0.84 (接近圆形)
|
||||
|
||||
#### Wheel (车轮)
|
||||
- 平均尺寸: 20×52 px
|
||||
- 长宽比: 0.42 (竖直椭圆)
|
||||
- 特点: 高度明显大于宽度
|
||||
|
||||
#### Plate (车牌)
|
||||
- 平均尺寸: 28×13 px
|
||||
- 长宽比: 2.25 (横向矩形)
|
||||
- 特点: 典型的车牌横向特征
|
||||
|
||||
---
|
||||
|
||||
## 5. 3D标注深度分析
|
||||
|
||||
### 5.1 深度分布统计
|
||||
|
||||
#### Vehicle (车辆) - 22,070,252个3D标注
|
||||
| 深度范围 | 对象数量 | 占比 |
|
||||
|---------|---------|------|
|
||||
| 0-10m | 2,666,042 | 12.1% |
|
||||
| 10-20m | 3,437,999 | 15.6% |
|
||||
| 20-30m | 3,158,252 | 14.3% |
|
||||
| 30-50m | 5,195,185 | 23.5% |
|
||||
| 50-80m | 4,885,009 | 22.1% |
|
||||
| 80-120m | 2,720,248 | 12.3% |
|
||||
| >120m | 7,517 | 0.03% |
|
||||
|
||||
- **平均深度**: 41.8m
|
||||
- **中位深度**: 36.1m
|
||||
- **95%分位**: 97.6m
|
||||
- **深度范围**: -8.5m ~ 128.9m
|
||||
|
||||
#### Pedestrian (行人) - 3,955,590个3D标注
|
||||
| 深度范围 | 对象数量 | 占比 |
|
||||
|---------|---------|------|
|
||||
| 0-10m | 626,552 | 15.8% |
|
||||
| 10-20m | 1,024,844 | 25.9% |
|
||||
| 20-30m | 787,225 | 19.9% |
|
||||
| 30-50m | 955,993 | 24.2% |
|
||||
| 50-80m | 491,938 | 12.4% |
|
||||
| 80-120m | 69,038 | 1.7% |
|
||||
| >120m | 0 | 0% |
|
||||
|
||||
- **平均深度**: 28.3m
|
||||
- **中位深度**: 23.7m
|
||||
- **最大深度**: 119.9m
|
||||
|
||||
#### Bicycle (自行车) - 6,368,673个3D标注
|
||||
| 深度范围 | 对象数量 | 占比 |
|
||||
|---------|---------|------|
|
||||
| 0-10m | 1,343,418 | 21.1% |
|
||||
| 10-20m | 1,841,284 | 28.9% |
|
||||
| 20-30m | 1,363,130 | 21.4% |
|
||||
| 30-50m | 1,247,522 | 19.6% |
|
||||
| 50-80m | 521,347 | 8.2% |
|
||||
| 80-120m | 51,972 | 0.8% |
|
||||
|
||||
- **平均深度**: 24.1m
|
||||
- **中位深度**: 20.0m
|
||||
|
||||
#### Rider (骑行者) - 5,490,105个3D标注
|
||||
| 深度范围 | 对象数量 | 占比 |
|
||||
|---------|---------|------|
|
||||
| 0-10m | 776,346 | 14.1% |
|
||||
| 10-20m | 1,145,211 | 20.9% |
|
||||
| 20-30m | 896,225 | 16.3% |
|
||||
| 30-50m | 1,349,814 | 24.6% |
|
||||
| 50-80m | 990,871 | 18.0% |
|
||||
| 80-120m | 331,515 | 6.0% |
|
||||
| >120m | 123 | 0.002% |
|
||||
|
||||
- **平均深度**: 34.6m
|
||||
- **中位深度**: 29.1m
|
||||
|
||||
**深度分布特点**:
|
||||
- 车辆检测距离最远(均值41.8m),适合高速场景
|
||||
- 行人和自行车主要分布在近距离(<30m占比>60%)
|
||||
- 骑行者深度分布介于车辆和行人之间
|
||||
- 30-50m是所有类别的关键检测范围
|
||||
|
||||
---
|
||||
|
||||
## 6. 3D物体尺寸分析
|
||||
|
||||
### 6.1 真实世界物理尺寸(米)
|
||||
|
||||
#### Vehicle (车辆)
|
||||
| 维度 | 长度(m) | 高度(m) | 宽度(m) |
|
||||
|-----|--------|--------|--------|
|
||||
| 均值 | 4.89 | 1.82 | 1.99 |
|
||||
| 中位数 | 4.40 | 1.63 | 1.93 |
|
||||
| 5%-95%范围 | 4.11-8.52 | 1.50-3.29 | 1.81-2.75 |
|
||||
|
||||
**分析**:
|
||||
- 中位长度4.40m,符合轿车尺寸
|
||||
- 高度1.63m,典型轿车/SUV高度
|
||||
- 长度95%分位8.52m,包含货车/客车
|
||||
|
||||
#### Pedestrian (行人)
|
||||
| 维度 | 长度(m) | 高度(m) | 宽度(m) |
|
||||
|-----|--------|--------|--------|
|
||||
| 均值 | 0.63 | 1.67 | 0.66 |
|
||||
| 中位数 | 0.62 | 1.68 | 0.65 |
|
||||
| 标准差 | 0.03 | 0.05 | 0.04 |
|
||||
|
||||
**分析**:
|
||||
- 高度1.67m,符合成人平均身高
|
||||
- 尺寸标准差极小,标注一致性好
|
||||
|
||||
#### Bicycle (自行车)
|
||||
| 维度 | 长度(m) | 高度(m) | 宽度(m) |
|
||||
|-----|--------|--------|--------|
|
||||
| 均值 | 1.60 | 1.20 | 0.66 |
|
||||
| 中位数 | 1.59 | 1.18 | 0.65 |
|
||||
|
||||
**分析**:
|
||||
- 长度1.60m,典型自行车尺寸
|
||||
- 宽度0.66m,标准车把宽度
|
||||
|
||||
#### Rider (骑行者)
|
||||
| 维度 | 长度(m) | 高度(m) | 宽度(m) |
|
||||
|-----|--------|--------|--------|
|
||||
| 均值 | 1.74 | 1.57 | 0.83 |
|
||||
| 中位数 | 1.73 | 1.58 | 0.83 |
|
||||
|
||||
**分析**:
|
||||
- 比自行车更高(1.57m vs 1.20m)
|
||||
- 宽度更大(0.83m vs 0.66m),包含人体
|
||||
|
||||
---
|
||||
|
||||
## 7. 姿态与朝向分析
|
||||
|
||||
### 7.1 航向角(rot_y)分布
|
||||
|
||||
| 类别 | 均值(rad) | 标准差 | 中位数 |
|
||||
|-----|----------|-------|-------|
|
||||
| Vehicle | -0.34 | 1.69 | -1.49 |
|
||||
| Pedestrian | -0.07 | 1.79 | -0.11 |
|
||||
| Bicycle | -0.16 | 2.10 | -0.03 |
|
||||
| Rider | -0.34 | 1.69 | -1.39 |
|
||||
|
||||
**解读**:
|
||||
- 所有类别均值接近0,朝向分布较为均衡
|
||||
- 标准差较大(~1.7-2.1),覆盖各个方向
|
||||
- 车辆和骑行者中位数-1.4rad,可能反映主要交通流向
|
||||
|
||||
### 7.2 观测角(alpha)分布
|
||||
|
||||
| 类别 | 均值(rad) | 标准差 | 中位数 |
|
||||
|-----|----------|-------|-------|
|
||||
| Vehicle | -0.30 | 1.76 | -1.24 |
|
||||
| Pedestrian | -0.10 | 1.77 | -0.30 |
|
||||
| Bicycle | 0.37 | 1.96 | 0.44 |
|
||||
| Rider | -0.28 | 1.81 | -1.29 |
|
||||
|
||||
### 7.3 横向位置(lateral_x3d)分布
|
||||
|
||||
| 类别 | 均值(m) | 标准差(m) | 中位数(m) |
|
||||
|-----|--------|----------|----------|
|
||||
| Vehicle | -2.56 | 13.21 | -2.56 |
|
||||
| Pedestrian | 0.10 | 12.35 | 3.11 |
|
||||
| Bicycle | 0.08 | 11.66 | 2.80 |
|
||||
| Rider | -0.87 | 11.47 | 0.78 |
|
||||
|
||||
**分析**:
|
||||
- 车辆中位数-2.56m,偏向左侧(可能为对向车道)
|
||||
- 行人和自行车中位数为正,偏向右侧(路边)
|
||||
- 标准差10-13m,横向分布较广
|
||||
|
||||
---
|
||||
|
||||
## 8. 车辆面可见性分析
|
||||
|
||||
### 8.1 面可见性统计(仅车辆类别)
|
||||
|
||||
| 面向 | 可见率 | 平均分数 | 中位分数 |
|
||||
|-----|-------|---------|---------|
|
||||
| 前面 (front) | 93.2% | 0.318 | 0.000 |
|
||||
| 后面 (rear) | 90.8% | 0.510 | 0.698 |
|
||||
| 左侧 (left) | 68.1% | 0.241 | 0.145 |
|
||||
| 右侧 (right) | 68.1% | 0.148 | 0.000 |
|
||||
|
||||
**关键发现**:
|
||||
- 前后面可见率>90%,侧面可见率~68%
|
||||
- 后面平均得分最高(0.510),可能因尾灯、车牌更易识别
|
||||
- 右侧得分最低(0.148),可能受摄像头安装位置影响
|
||||
- 中位分数显示: 后面>左侧>前面≈右侧
|
||||
|
||||
---
|
||||
|
||||
## 9. 数据质量评估
|
||||
|
||||
### 9.1 Cut类型分布
|
||||
|
||||
| 类型 | 对象数量 | 占比 |
|
||||
|-----|---------|------|
|
||||
| noncut (完整) | 20,252,105 | 91.6% |
|
||||
| cut_in (切入) | 1,175,262 | 5.3% |
|
||||
| cut_out (切出) | 642,885 | 2.9% |
|
||||
|
||||
**总计**: 22,070,252个有3D标注的对象
|
||||
|
||||
### 9.2 数据质量问题
|
||||
|
||||
| 问题类型 | 数量 | 占比 |
|
||||
|---------|-----|------|
|
||||
| 缺失3D标注 | 23,122,502 | 18.1% |
|
||||
| 深度≤0(无效) | 19,236 | 0.02% |
|
||||
| 深度>200m(异常) | 0 | 0% |
|
||||
| BBox<8px(过小) | 21,474,341 | 16.8% |
|
||||
|
||||
**质量评估**:
|
||||
- ✅ **优秀**: 无超远距离异常值(>200m)
|
||||
- ✅ **良好**: 负深度值极少(19,236个,0.02%)
|
||||
- ⚠️ **注意**: 18.1%对象缺失3D标注(主要为辅助类别)
|
||||
- ⚠️ **注意**: 16.8%对象BBox过小(<8px),可能影响训练
|
||||
|
||||
---
|
||||
|
||||
## 10. 数据集特点总结
|
||||
|
||||
### 10.1 优势
|
||||
1. **数据规模大**: 260万+图像,1.28亿+标注对象
|
||||
2. **场景复杂**: 平均每图48.5个对象,涵盖13个类别
|
||||
3. **标注完整**: 100%图像有标签,无缺失
|
||||
4. **深度覆盖广**: 车辆0-120m全范围覆盖
|
||||
5. **3D一致性好**: 物体尺寸标准差小,标注质量高
|
||||
6. **朝向分布均衡**: 各个方向角度覆盖完整
|
||||
|
||||
### 10.2 局限性
|
||||
1. **3D标注不全**: 仅4/13类别有3D标注
|
||||
2. **小目标多**: 16.8%的BBox<8px,可能影响检测精度
|
||||
3. **类别不平衡**: 车辆占28%,某些类别<1%
|
||||
4. **切边对象**: 8.2%的对象存在cut_in/cut_out情况
|
||||
|
||||
### 10.3 应用建议
|
||||
|
||||
#### 模型训练
|
||||
- **多尺度训练**: 应对10px-1900px的尺寸变化
|
||||
- **深度分段策略**:
|
||||
- 近距离(<30m): 重点优化精度
|
||||
- 中距离(30-80m): 平衡精度与召回
|
||||
- 远距离(>80m): 提升召回率
|
||||
- **数据增强**: 针对小目标(<8px)加强
|
||||
- **类别平衡**: 对低频类别采用重采样
|
||||
|
||||
#### 评估指标
|
||||
- 按深度分段评估AP
|
||||
- 按对象尺寸分段评估
|
||||
- 重点关注30-50m范围性能
|
||||
|
||||
#### 特殊处理
|
||||
- Cut对象(8.2%): 建立专门评估集
|
||||
- 遮挡处理: 利用面可见性分数
|
||||
- 小目标: 考虑FPN或多尺度融合
|
||||
|
||||
---
|
||||
|
||||
## 11. 数据集扩充建议
|
||||
|
||||
基于当前数据集的统计分析,针对模型性能提升和场景覆盖完整性,提出以下数据扩充建议:
|
||||
|
||||
### 11.1 类别均衡优化
|
||||
|
||||
#### 🎯 优先级1:低频类别数据增强
|
||||
|
||||
当前类别分布极不均衡,建议扩充以下低频类别:
|
||||
|
||||
| 类别 | 当前数量 | 占比 | 建议目标 | 扩充倍数 |
|
||||
|-----|---------|------|---------|---------|
|
||||
| tl_num (交通灯数字) | 696,538 | 0.5% | 2,000,000+ | 3x |
|
||||
| tl_wick (交通灯灯芯) | 3,611,666 | 2.8% | 6,000,000+ | 1.7x |
|
||||
| tsr (交通标志) | 4,957,683 | 3.9% | 8,000,000+ | 1.6x |
|
||||
| guideboard (指示牌) | 6,457,611 | 5.0% | 10,000,000+ | 1.5x |
|
||||
|
||||
**扩充策略**:
|
||||
- 专门收集城市路口、高速公路、园区等交通信号设施密集场景
|
||||
- 增加夜间、雨天等特殊光照条件下的交通灯数据
|
||||
- 补充不同类型交通标志(限速、禁令、警告等)
|
||||
- 采集不同语言、格式的指示牌样本
|
||||
|
||||
#### 🎯 优先级2:次要类别定向采集
|
||||
|
||||
| 类别 | 当前数量 | 建议扩充场景 |
|
||||
|-----|---------|------------|
|
||||
| tl_border | 5,645,436 | 复杂路口、多车道场景 |
|
||||
| roadblock | 8,058,184 | 施工区域、临时路障、交通管制 |
|
||||
| head | 14,610,038 | 密集人群、学校区域、商圈 |
|
||||
| wheel | 12,895,577 | 大型车辆(货车、客车)轮胎 |
|
||||
| plate | 9,939,105 | 不同国家/地区车牌样式 |
|
||||
|
||||
### 11.2 3D标注扩充计划
|
||||
|
||||
#### 🎯 优先级1:当前有3D标注类别的深度补充
|
||||
|
||||
**Vehicle (车辆)**
|
||||
```
|
||||
当前问题:
|
||||
- 超近距离(<5m): 仅占3.6% → 建议扩充到10%
|
||||
- 超远距离(>80m): 仅占12.3% → 建议特别关注80-120m范围
|
||||
|
||||
扩充场景:
|
||||
✓ 停车场近距离场景(1-5m)
|
||||
✓ 高速公路远距离场景(80-150m)
|
||||
✓ 大型车辆(货车、客车、工程车)
|
||||
✓ 特殊车辆(摩托车、三轮车、农用车)
|
||||
```
|
||||
|
||||
**Pedestrian (行人)**
|
||||
```
|
||||
当前问题:
|
||||
- 深度>80m的行人仅69,038个(1.7%)
|
||||
- 缺少远距离、小尺寸行人样本
|
||||
|
||||
扩充场景:
|
||||
✓ 远距离行人(50-100m)
|
||||
✓ 特殊姿态(蹲、躺、骑)
|
||||
✓ 不同年龄段(儿童、老人)
|
||||
✓ 携带大型物品的行人
|
||||
```
|
||||
|
||||
**Bicycle & Rider (自行车与骑行者)**
|
||||
```
|
||||
当前问题:
|
||||
- 深度>50m样本较少(bicycle: 8.2%, rider: 24.0%)
|
||||
- Rider的3D标注率82.5%,但Bicycle仅59.8%
|
||||
|
||||
扩充需求:
|
||||
✓ 平衡Bicycle和Rider的3D标注率
|
||||
✓ 增加电动车、滑板车等新型交通工具
|
||||
✓ 补充远距离骑行样本
|
||||
```
|
||||
|
||||
#### 🎯 优先级2:新类别3D标注添加
|
||||
|
||||
建议为以下类别补充3D标注(按优先级排序):
|
||||
|
||||
1. **roadblock (路障)** - 8,058,184个对象
|
||||
- 重要性:影响路径规划和避障决策
|
||||
- 3D信息需求:位置、高度、宽度
|
||||
- 建议标注率:50%以上
|
||||
|
||||
2. **guideboard (指示牌)** - 6,457,611个对象
|
||||
- 重要性:导航和车道保持
|
||||
- 3D信息需求:位置、朝向、高度
|
||||
- 建议标注率:30%以上
|
||||
|
||||
3. **tsr (交通标志)** - 4,957,683个对象
|
||||
- 重要性:合规驾驶决策
|
||||
- 3D信息需求:距离、位置
|
||||
- 建议标注率:40%以上
|
||||
|
||||
### 11.3 深度分布优化
|
||||
|
||||
#### 各类别深度扩充建议
|
||||
|
||||
```
|
||||
Vehicle:
|
||||
├─ 0-10m: 12.1% → 目标15% (+77.8万样本)
|
||||
├─ 10-20m: 15.6% → ✓ 已充足
|
||||
├─ 20-30m: 14.3% → ✓ 已充足
|
||||
├─ 30-50m: 23.5% → ✓ 已充足(主要区域)
|
||||
├─ 50-80m: 22.1% → ✓ 已充足
|
||||
└─ 80-120m: 12.3% → 目标15% (+63.1万样本)
|
||||
|
||||
Pedestrian:
|
||||
├─ 0-10m: 15.8% → ✓ 已充足
|
||||
├─ 10-20m: 25.9% → ✓ 已充足(主要区域)
|
||||
├─ 20-30m: 19.9% → ✓ 已充足
|
||||
├─ 30-50m: 24.2% → ✓ 已充足
|
||||
└─ 50-80m: 12.4% → 目标18% (+21.8万样本)
|
||||
80-120m: 1.7% → 目标5% (+13.1万样本)
|
||||
|
||||
Bicycle:
|
||||
├─ 0-10m: 21.1% → ✓ 已充足
|
||||
├─ 10-20m: 28.9% → ✓ 已充足(主要区域)
|
||||
└─ 50-120m: 9.0% → 目标15% (+38.2万样本)
|
||||
|
||||
Rider:
|
||||
├─ 各区域分布相对均衡 ✓
|
||||
└─ 建议维持当前分布比例
|
||||
```
|
||||
|
||||
### 11.4 小目标问题解决方案
|
||||
|
||||
**当前情况**:21,474,341个对象(16.8%)的BBox面积<8px
|
||||
|
||||
#### 策略A:高分辨率数据采集
|
||||
- 当前可能分辨率:1920×1080或更低
|
||||
- 建议扩充:2K (2560×1440) 或 4K (3840×2160)
|
||||
- 目标:将<8px占比降至10%以下
|
||||
|
||||
#### 策略B:近距离场景补充
|
||||
```
|
||||
重点采集场景:
|
||||
✓ 停车场近距离场景(1-5m)
|
||||
✓ 低速城区道路(<40km/h)
|
||||
✓ 拥堵路况(车辆密集,近距离)
|
||||
✓ 人行道近距离行人
|
||||
```
|
||||
|
||||
#### 策略C:数据增强策略
|
||||
```python
|
||||
针对小目标的训练增强:
|
||||
- Copy-Paste小目标增强
|
||||
- Mosaic数据增强倾斜(更多小目标)
|
||||
- 多尺度训练权重调整
|
||||
- FPN/PANET特征融合优化
|
||||
```
|
||||
|
||||
### 11.5 场景标签补充建议
|
||||
|
||||
#### 建议添加的场景元数据标签
|
||||
|
||||
**1. 天气条件** (当前缺失)
|
||||
```yaml
|
||||
weather:
|
||||
- sunny: 晴天
|
||||
- cloudy: 多云
|
||||
- rainy: 雨天 (小雨/中雨/大雨)
|
||||
- foggy: 雾天
|
||||
- snowy: 雪天
|
||||
|
||||
优先扩充:
|
||||
- 雨天数据(建议占比20-30%)
|
||||
- 雾天数据(建议占比5-10%)
|
||||
- 夜间降雨组合场景
|
||||
```
|
||||
|
||||
**2. 光照条件** (当前缺失)
|
||||
```yaml
|
||||
lighting:
|
||||
- daytime: 白天
|
||||
- morning: 早晨 (6:00-10:00)
|
||||
- noon: 中午 (10:00-16:00)
|
||||
- afternoon: 下午 (16:00-19:00)
|
||||
- nighttime: 夜间
|
||||
- dusk: 黄昏 (19:00-21:00)
|
||||
- night: 深夜 (21:00-6:00)
|
||||
- tunnel: 隧道
|
||||
|
||||
优先扩充:
|
||||
- 夜间数据(建议占比30-40%)
|
||||
- 黄昏/黎明过渡时段(建议占比10%)
|
||||
- 隧道进出场景(建议占比5%)
|
||||
```
|
||||
|
||||
**3. 道路类型** (当前缺失)
|
||||
```yaml
|
||||
road_type:
|
||||
- highway: 高速公路
|
||||
- urban_road: 城市道路
|
||||
- suburb_road: 郊区道路
|
||||
- parking_lot: 停车场
|
||||
- residential: 住宅区
|
||||
- industrial: 工业区
|
||||
|
||||
优先扩充:
|
||||
- 停车场场景(近距离、多角度)
|
||||
- 住宅区(复杂交通参与者)
|
||||
- 高速公路(高速、远距离)
|
||||
```
|
||||
|
||||
**4. 交通密度** (当前缺失)
|
||||
```yaml
|
||||
traffic_density:
|
||||
- sparse: 稀疏 (<10 objects/image)
|
||||
- medium: 中等 (10-50 objects/image)
|
||||
- dense: 密集 (50-100 objects/image)
|
||||
- jam: 拥堵 (>100 objects/image)
|
||||
|
||||
当前分布:
|
||||
- sparse: ~5%分位 (8 objects)
|
||||
- medium: 25-75分位 (25-66 objects)
|
||||
- dense: 75-95分位 (66-103 objects)
|
||||
- jam: >95分位 (>103 objects)
|
||||
|
||||
建议:增加极端场景
|
||||
- 空旷场景(目标<5)
|
||||
- 极度拥堵(目标>150)
|
||||
```
|
||||
|
||||
**5. 遮挡程度** (当前缺失)
|
||||
```yaml
|
||||
occlusion_level:
|
||||
- none: 无遮挡 (0-10%)
|
||||
- partial: 部分遮挡 (10-50%)
|
||||
- heavy: 严重遮挡 (50-80%)
|
||||
- extreme: 极度遮挡 (>80%)
|
||||
|
||||
建议采集:
|
||||
- 多车并行场景(侧面遮挡)
|
||||
- 大车遮挡小车/行人
|
||||
- 护栏、植被遮挡
|
||||
- 雨刷、车身部分遮挡
|
||||
```
|
||||
|
||||
**6. 特殊场景标签**
|
||||
```yaml
|
||||
special_scenarios:
|
||||
- construction_zone: 施工区域
|
||||
- accident_scene: 事故现场
|
||||
- school_zone: 学校区域
|
||||
- crosswalk: 人行横道
|
||||
- intersection: 十字路口
|
||||
- roundabout: 环岛
|
||||
- merge_lane: 汇入车道
|
||||
- toll_station: 收费站
|
||||
|
||||
优先扩充:
|
||||
- 施工区域(路障、锥桶)
|
||||
- 学校区域(儿童、家长、慢速)
|
||||
- 复杂路口(多方向、多目标)
|
||||
```
|
||||
|
||||
### 11.6 边缘情况与困难样本
|
||||
|
||||
#### Cut对象扩充(当前8.2%)
|
||||
|
||||
```
|
||||
当前分布:
|
||||
- noncut: 91.6% (20,252,105)
|
||||
- cut_in: 5.3% (1,175,262)
|
||||
- cut_out: 2.9% (642,885)
|
||||
|
||||
建议目标:
|
||||
- cut_in: 扩充到8-10% (+60万样本)
|
||||
- cut_out: 扩充到5-6% (+45万样本)
|
||||
|
||||
采集重点:
|
||||
✓ 边缘车辆突然切入
|
||||
✓ 部分在frame外的行人
|
||||
✓ 图像边缘的骑行者
|
||||
✓ 半身出现的大型车辆
|
||||
```
|
||||
|
||||
#### 困难样本专项采集
|
||||
|
||||
**1. 遮挡场景**
|
||||
- 车辆互相遮挡(并排行驶)
|
||||
- 大型车辆遮挡后方小车
|
||||
- 护栏、树木遮挡
|
||||
- 行人/骑行者被车辆遮挡
|
||||
|
||||
**2. 极端光照**
|
||||
- 强逆光(太阳直射镜头)
|
||||
- 夜间远光灯眩光
|
||||
- 隧道进出口光线突变
|
||||
- 玻璃反射/水面倒影
|
||||
|
||||
**3. 相似目标混淆**
|
||||
- 骑行者 vs 电动车骑手
|
||||
- 自行车 vs 摩托车
|
||||
- 小客车 vs SUV
|
||||
- 行人 vs 路边广告牌人像
|
||||
|
||||
**4. 特殊姿态与状态**
|
||||
- 侧翻/事故车辆
|
||||
- 蹲坐/倚靠的行人
|
||||
- 推行自行车的人
|
||||
- 多人拥挤场景
|
||||
|
||||
### 11.7 数据采集优先级矩阵
|
||||
|
||||
| 优先级 | 扩充内容 | 预期样本量 | 预期效果 |
|
||||
|-------|---------|-----------|---------|
|
||||
| ⭐⭐⭐⭐⭐ | 夜间数据 | +80万图像 | 提升夜间AP 15-20% |
|
||||
| ⭐⭐⭐⭐⭐ | 雨天/雾天数据 | +50万图像 | 提升恶劣天气鲁棒性 |
|
||||
| ⭐⭐⭐⭐⭐ | 交通灯/标志3D标注 | +500万3D标注 | 提升决策可靠性 |
|
||||
| ⭐⭐⭐⭐ | 远距离样本(>80m) | +100万对象 | 提升远距离召回率 |
|
||||
| ⭐⭐⭐⭐ | 近距离样本(<5m) | +80万对象 | 提升近距离精度 |
|
||||
| ⭐⭐⭐⭐ | 低频类别均衡 | +300万对象 | 平衡各类别性能 |
|
||||
| ⭐⭐⭐ | 遮挡场景 | +30万图像 | 提升遮挡处理能力 |
|
||||
| ⭐⭐⭐ | 边缘Cut场景 | +100万对象 | 提升边缘检测 |
|
||||
| ⭐⭐ | 高分辨率数据 | +20万图像 | 改善小目标检测 |
|
||||
| ⭐⭐ | 特殊场景标签 | 全量标注 | 支持场景自适应 |
|
||||
|
||||
### 11.8 数据采集实施计划
|
||||
|
||||
#### 阶段1:短期扩充(1-3个月)
|
||||
```
|
||||
目标:解决最紧迫问题
|
||||
1. 夜间数据采集(80万图像)
|
||||
2. 低频类别定向采集(200万对象)
|
||||
3. 远近距离补充(100万对象)
|
||||
4. 基础场景标签标注(100万图像)
|
||||
|
||||
预期提升:整体mAP +3-5%
|
||||
```
|
||||
|
||||
#### 阶段2:中期优化(3-6个月)
|
||||
```
|
||||
目标:全面质量提升
|
||||
1. 恶劣天气数据(50万图像)
|
||||
2. 3D标注扩展到新类别(500万标注)
|
||||
3. 困难样本专项采集(30万图像)
|
||||
4. 完整场景元数据标注(全量)
|
||||
|
||||
预期提升:整体mAP +5-8%
|
||||
```
|
||||
|
||||
#### 阶段3:长期完善(6-12个月)
|
||||
```
|
||||
目标:构建世界级数据集
|
||||
1. 边缘场景覆盖(corner cases)
|
||||
2. 多地域、多国家数据
|
||||
3. 高分辨率升级(4K)
|
||||
4. 时序数据采集(视频片段)
|
||||
|
||||
预期提升:整体mAP +8-12%
|
||||
```
|
||||
|
||||
### 11.9 数据质量控制建议
|
||||
|
||||
#### 采集质量标准
|
||||
```yaml
|
||||
硬件要求:
|
||||
- 分辨率: ≥1920×1080 (建议2K)
|
||||
- 帧率: ≥15 FPS
|
||||
- 动态范围: HDR优先
|
||||
- 镜头畸变: <5%
|
||||
|
||||
标注质量标准:
|
||||
- BBox IoU一致性: >0.85
|
||||
- 3D深度误差: <10%
|
||||
- 朝向角误差: <15°
|
||||
- 类别标注准确率: >98%
|
||||
```
|
||||
|
||||
#### 质量验证流程
|
||||
1. 自动化检查(异常值、格式错误)
|
||||
2. 随机抽样人工复核(5-10%)
|
||||
3. 交叉验证(不同标注员)
|
||||
4. 模型验证(训练效果验证)
|
||||
|
||||
---
|
||||
|
||||
## 12. 数据统计来源
|
||||
|
||||
- **分析模式**: original
|
||||
- **统计时间**: 基于完整训练集
|
||||
- **3D标注格式**: 包含深度、尺寸、朝向、位置等信息
|
||||
- **统计方法**: 基于百分位数的鲁棒统计
|
||||
|
||||
---
|
||||
|
||||
*本报告由自动化数据分析工具生成,统计数据来源于dataset_profiling_results_original/train/summary.json*
|
||||
372
tools/scripts_for_gt/profiling/DATASET_PROFILING_README.md
Executable file
372
tools/scripts_for_gt/profiling/DATASET_PROFILING_README.md
Executable file
@@ -0,0 +1,372 @@
|
||||
# Dataset Profiling — YOLOv5-3D 数据画像统计分析工具
|
||||
|
||||
## 概述
|
||||
|
||||
`dataset_profiling.py` 是一个针对 YOLOv5-3D 训练和评测数据集的综合统计分析工具,能够自动生成数据集的完整"数据画像"(Data Portrait),包含文本报告、JSON 结构化数据和可视化图表。
|
||||
|
||||
---
|
||||
|
||||
## 功能模块
|
||||
|
||||
| # | 分析模块 | 说明 |
|
||||
|---|---------|------|
|
||||
| 1 | **数据集概览** | 图片/标签数量、标签格式分布(50列/18列/6列) |
|
||||
| 2 | **类别统计** | 每类目标数量、占比、2D/3D 覆盖度 |
|
||||
| 3 | **2D 框分析** | 尺寸分布、宽高比、位置热力图 |
|
||||
| 4 | **3D 几何分析** | 深度、三维尺寸(L×H×W)、旋转角分布 |
|
||||
| 5 | **面可见性分析** | 四面可见率、置信度分布(车辆/三轮车) |
|
||||
| 6 | **截断类型分析** | cut_in / cut_out / noncut / partial_cut 分布 |
|
||||
| 7 | **图像密度分析** | 每张图像的目标数、类别共现统计 |
|
||||
| 8 | **数据质量检查** | NaN 字段、无效深度、异常深度、微小框检测 |
|
||||
| 9 | **训练/验证对比** | 类别比例、深度分布、尺寸分布的 Train vs Val 对比 |
|
||||
|
||||
---
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
# 基于原图的完整分析(默认模式)
|
||||
python dataset_profiling.py
|
||||
|
||||
# 基于 ROI 的分析(匹配训练流水线)
|
||||
python dataset_profiling.py --analysis-mode roi
|
||||
|
||||
# 仅分析验证集,限制 5000 张图
|
||||
python dataset_profiling.py --split val --max-files 5000
|
||||
|
||||
# 同时指定模式和输出目录
|
||||
python dataset_profiling.py --analysis-mode roi --split both --output-dir my_profiling
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 命令行参数
|
||||
|
||||
| 参数 | 类型 | 默认值 | 说明 |
|
||||
|------|------|--------|------|
|
||||
| `--data` | str | `data/mono3d.yaml` | 数据集配置 YAML 文件路径 |
|
||||
| `--split` | str | `both` | 分析哪个数据集划分:`train`、`val` 或 `both` |
|
||||
| `--max-files` | int | `0` | 每个 split 最多处理的图片数(0 = 全部) |
|
||||
| `--output-dir` | str | `dataset_profiling_results` | 输出目录(自动追加 `_original` 或 `_roi` 后缀) |
|
||||
| `--img-size` | int int | 自动 | 像素单位转换尺寸 `[宽 高]`;original 模式默认取 `ori_img_size`,roi 模式默认取 `roi` |
|
||||
| `--analysis-mode` | str | `original` | 分析模式:`original`(原图坐标系)或 `roi`(ROI 坐标系) |
|
||||
|
||||
---
|
||||
|
||||
## 分析模式详解
|
||||
|
||||
### `original` 模式(默认)
|
||||
|
||||
直接使用标签文件中的归一化坐标,基于原始图像尺寸(默认 1920×1080)转换为像素单位进行统计。不做任何 ROI 裁剪或过滤。
|
||||
|
||||
适用场景:
|
||||
- 了解原始标注数据的整体分布
|
||||
- 检查标注质量和覆盖范围
|
||||
- 不依赖标定参数的快速分析
|
||||
|
||||
### `roi` 模式
|
||||
|
||||
模拟训练流水线的 ROI 处理逻辑,从标定文件计算每个相机的实际 ROI 区域,对标签进行过滤和截断:
|
||||
|
||||
```
|
||||
1. 读取相机标定参数(camera4.json)
|
||||
2. 计算消失点:vanish_y = cv - fy × tan(pitch × π/180)
|
||||
3. 计算 ROI 边界:以 (oriW/2, vanish_y) 为中心裁剪 roi_size 区域
|
||||
4. 过滤完全在 ROI 外的目标
|
||||
5. 截断跨越 ROI 边界的目标(设置 cut_in/cut_out)
|
||||
6. 将坐标重新归一化到 ROI 空间
|
||||
```
|
||||
|
||||
适用场景:
|
||||
- 分析**模型实际训练时**看到的数据分布
|
||||
- 评估 ROI 裁剪对数据量的影响
|
||||
- 对比原图 vs ROI 的分布差异
|
||||
|
||||
#### ROI 计算流程图
|
||||
|
||||
```
|
||||
原始图像 (1920×1080)
|
||||
│
|
||||
▼
|
||||
读取 camera4.json
|
||||
(focal_v, cv, pitch)
|
||||
│
|
||||
▼
|
||||
vanish_y = cv - fy × tan(pitch × π/180)
|
||||
│
|
||||
▼
|
||||
ROI 中心 = (960, vanish_y)
|
||||
ROI 区域 = [roi_x1, roi_y1, roi_x2, roi_y2] (1920×960)
|
||||
│
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ 原始标签 │
|
||||
│ (归一化坐标) │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌────┴────┐
|
||||
▼ ▼
|
||||
ROI 内 ROI 外 → 过滤丢弃
|
||||
│
|
||||
├─ 完全在内 → 直接重归一化
|
||||
│
|
||||
└─ 部分跨界 → clip截断 + 设置cut类型
|
||||
│
|
||||
▼
|
||||
ROI 空间归一化坐标
|
||||
```
|
||||
|
||||
#### 标定文件路径推导
|
||||
|
||||
```
|
||||
图像路径: .../sequence_name/images/xxx.png
|
||||
标定路径: .../sequence_name/calib/L2_calib/camera4.json
|
||||
```
|
||||
|
||||
同一序列目录下所有图像共享同一份标定,脚本会自动按序列目录缓存,避免重复读取。
|
||||
|
||||
---
|
||||
|
||||
## 输出文件说明
|
||||
|
||||
运行后在输出目录下按 split 组织,结构如下:
|
||||
|
||||
```
|
||||
dataset_profiling_results_roi/
|
||||
├── train/
|
||||
│ ├── report.txt # 文本格式完整报告
|
||||
│ ├── summary.json # JSON 格式结构化数据
|
||||
│ ├── 01_overview.png # 数据集概览(类别分布 + 格式分布 + 密度统计)
|
||||
│ ├── 02_bbox2d_analysis.png # 2D 框分析(尺寸散点 + 面积分布 + 位置热力图)
|
||||
│ ├── 03_depth_analysis.png # 深度分布(直方图 + CDF)
|
||||
│ ├── 04_dimensions_3d.png # 3D 尺寸分布(L/H/W 直方图)
|
||||
│ ├── 05_rotation_analysis.png # 旋转角分布(rot_y + alpha)
|
||||
│ ├── 06_face_visibility_cut.png# 面可见性 + 截断类型分布
|
||||
│ ├── 07_bev_scatter.png # 鸟瞰图散点(X3D vs Z3D)
|
||||
│ └── 08_depth_vs_area.png # 深度-面积关系图
|
||||
├── val/
|
||||
│ └── ... (同上)
|
||||
└── 09_train_vs_val.png # Train vs Val 对比(类别比例 + 深度 + 尺寸)
|
||||
```
|
||||
|
||||
### 可视化图表详解
|
||||
|
||||
| 图表 | 内容 | 用途 |
|
||||
|------|------|------|
|
||||
| `01_overview` | 类别数量条形图、标签格式饼图、每图目标数统计 | 快速了解数据集规模和组成 |
|
||||
| `02_bbox2d_analysis` | Top6 类别的 W-H 散点图、面积直方图、中心位置热力图 | 分析目标大小和空间分布 |
|
||||
| `03_depth_analysis` | 每个 3D 类别的深度直方图(含均值/中位数标线)和 CDF 曲线 | 分析目标距离分布 |
|
||||
| `04_dimensions_3d` | 每个 3D 类别的 Length/Height/Width 直方图(含均值±标准差) | 分析目标物理尺寸 |
|
||||
| `05_rotation_analysis` | rot_y 和 alpha 的角度直方图(标注前/后/左/右方位) | 分析朝向分布 |
|
||||
| `06_face_visibility_cut` | 四面可见率条形图 + cut 类型饼图 | 分析面标注完整度和截断情况 |
|
||||
| `07_bev_scatter` | X-Z 鸟瞰散点图 | 直观展示目标空间分布 |
|
||||
| `08_depth_vs_area` | 深度 vs 2D 框面积散点图 | 验证深度-尺寸关系合理性 |
|
||||
| `09_train_vs_val` | 类别比例、深度、尺寸的 Train/Val 双柱对比 | 检查训练集和验证集分布一致性 |
|
||||
|
||||
---
|
||||
|
||||
## 标签格式说明
|
||||
|
||||
脚本支持三种原始标签格式,统一解析为 **47 维内部表示**:
|
||||
|
||||
### 50 列格式(vehicle / tricycle)
|
||||
|
||||
带四面 3D 标注的完整格式:
|
||||
|
||||
```
|
||||
cls x y w h x3d y3d z3d l h w rot_y xc yc ? alpha ? face_front(8)... face_rear(8)... face_left(8)... face_right(8)...
|
||||
```
|
||||
|
||||
每个 face 包含 8 个值:`[x3d, y3d, z3d, alpha, xc, yc, score, is_visible]`
|
||||
|
||||
### 18 列格式(pedestrian / bicycle / rider)
|
||||
|
||||
带 3D 信息但无面标注:
|
||||
|
||||
```
|
||||
cls x y w h x3d y3d z3d l h w rot_y xc yc ? alpha ? ...
|
||||
```
|
||||
|
||||
### 6 列格式(2D-only 类别)
|
||||
|
||||
仅有 2D 框信息:
|
||||
|
||||
```
|
||||
cls x y w h conf
|
||||
```
|
||||
|
||||
### 47 维内部表示索引表
|
||||
|
||||
| 索引 | 字段 | 说明 |
|
||||
|------|------|------|
|
||||
| 0 | cls | 类别 ID |
|
||||
| 1-4 | x, y, w, h | 2D 框(归一化 xywh) |
|
||||
| 5-7 | x3d, y3d, z3d | 3D 中心坐标(米) |
|
||||
| 8-10 | length, height, width | 3D 尺寸(米) |
|
||||
| 11 | rot_y | 全局旋转角(弧度) |
|
||||
| 12-13 | xc, yc | 3D 投影中心(归一化) |
|
||||
| 14 | alpha | 观察角(弧度) |
|
||||
| 15-22 | face_front | 前面 [x3d, y3d, z3d, alpha, xc, yc, score, is_visible] |
|
||||
| 23-30 | face_rear | 后面 |
|
||||
| 31-38 | face_left | 左面 |
|
||||
| 39-46 | face_right | 右面 |
|
||||
|
||||
---
|
||||
|
||||
## 支持的类别
|
||||
|
||||
| ID | 类别名 | 标签格式 | 3D 信息 |
|
||||
|----|--------|----------|---------|
|
||||
| 0 | vehicle | 50 列 | 四面 3D |
|
||||
| 1 | pedestrian | 18 列 | 整体 3D |
|
||||
| 2 | bicycle | 18 列 | 整体 3D |
|
||||
| 3 | rider | 18 列 | 整体 3D |
|
||||
| 4 | roadblock | 6 列 | 无 |
|
||||
| 5 | head | 6 列 | 无 |
|
||||
| 6 | tsr | 6 列 | 无 |
|
||||
| 7 | guideboard | 6 列 | 无 |
|
||||
| 8 | plate | 6 列 | 无 |
|
||||
| 9 | wheel | 6 列 | 无 |
|
||||
| 10 | tl_border | 6 列 | 无 |
|
||||
| 11 | tl_wick | 6 列 | 无 |
|
||||
| 12 | tl_num | 6 列 | 无 |
|
||||
| 13 | tricycle | 50 列 | 四面 3D |
|
||||
|
||||
---
|
||||
|
||||
## 截断类型(Cut Type)
|
||||
|
||||
当 ROI 裁剪导致车辆目标被部分截断时,根据旋转角 `rot_y` 判断截断类型:
|
||||
|
||||
| 类型 | 判定条件 | 面标注处理 |
|
||||
|------|---------|-----------|
|
||||
| **noncut** | 目标完全在 ROI 内,四面标注均有效 | 保持不变 |
|
||||
| **cut_in** | `rot_y ∈ [-π, 0]`(驶入方向) | 保留前面,其余面置为 -1 |
|
||||
| **cut_out** | `rot_y ∈ (0, π]`(驶出方向) | 保留后面,其余面置为 -1 |
|
||||
| **partial_cut** | 其他部分截断情况 | 根据实际可见性处理 |
|
||||
|
||||
---
|
||||
|
||||
## 数据质量指标
|
||||
|
||||
| 指标 | 说明 | 关注阈值 |
|
||||
|------|------|---------|
|
||||
| `nan_3d_objects` | 3D 类别目标但 3D 字段为 NaN | 应为 0 |
|
||||
| `invalid_depth_le0` | 深度 ≤ 0m 的目标 | 应尽量少 |
|
||||
| `outlier_depth_gt200m` | 深度 > 200m 的目标 | 可能为标注错误 |
|
||||
| `tiny_box_lt8px` | 2D 框宽或高 < 8px 的目标 | 训练时可能无法学习 |
|
||||
|
||||
---
|
||||
|
||||
## 数据配置(YAML)
|
||||
|
||||
脚本从 YAML 配置文件中读取以下关键字段:
|
||||
|
||||
```yaml
|
||||
# data/mono3d.yaml 示例
|
||||
path: /mnt/mono3d_nfs/xdzhu_data/Mono3d/Mono3d_4face_2m_g1m3
|
||||
|
||||
train:
|
||||
- /path/to/train_full.txt
|
||||
- /path/to/train_CNCAP.txt
|
||||
val:
|
||||
- /path/to/val_3000.txt
|
||||
|
||||
ori_img_size: [1920, 1080] # 原始图像尺寸 [W, H]
|
||||
roi: [1920, 960] # ROI 裁剪尺寸 [W, H]
|
||||
virtual_fx: 537 # 虚拟焦距(ROI 模式下的参考值)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 典型用例
|
||||
|
||||
### 1. 全量训练集画像
|
||||
|
||||
```bash
|
||||
python dataset_profiling.py --split train --analysis-mode roi
|
||||
```
|
||||
|
||||
### 2. 快速预览验证集
|
||||
|
||||
```bash
|
||||
python dataset_profiling.py --split val --max-files 3000 --analysis-mode original
|
||||
```
|
||||
|
||||
### 3. 对比 original vs roi 的分布差异
|
||||
|
||||
```bash
|
||||
# 分别运行两种模式
|
||||
python dataset_profiling.py --split val --analysis-mode original
|
||||
python dataset_profiling.py --split val --analysis-mode roi
|
||||
|
||||
# 输出分别在:
|
||||
# dataset_profiling_results_original/val/
|
||||
# dataset_profiling_results_roi/val/
|
||||
```
|
||||
|
||||
### 4. 自定义图像尺寸(模型输入分辨率)
|
||||
|
||||
```bash
|
||||
# 按模型输入尺寸 704×352 做像素统计
|
||||
python dataset_profiling.py --analysis-mode roi --img-size 704 352
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 代码架构
|
||||
|
||||
```
|
||||
dataset_profiling.py
|
||||
│
|
||||
├── 常量定义
|
||||
│ ├── CLASS_NAMES # 14 类名称映射
|
||||
│ ├── FACE_BASED_CLASSES # 面标注类别 {vehicle, tricycle}
|
||||
│ ├── ALL_3D_CLASSES # 所有 3D 类别
|
||||
│ └── IDX # 47 维内部索引表
|
||||
│
|
||||
├── 标签解析
|
||||
│ ├── parse_label_file() # 解析单个标签文件 → 47 维数组列表
|
||||
│ ├── image_path_to_label_path() # 图片路径 → 标签路径
|
||||
│ └── resolve_image_paths() # 从 YAML 配置解析全部图片路径
|
||||
│
|
||||
├── ROI 处理(--analysis-mode roi 时启用)
|
||||
│ ├── read_calib_for_image() # 读取相机标定 camera4.json
|
||||
│ ├── compute_roi_bounds() # 从标定计算 ROI 边界
|
||||
│ ├── get_roi_for_image() # 按序列缓存的 ROI 查询
|
||||
│ └── filter_clip_labels_to_roi() # 过滤/截断/重归一化标签
|
||||
│
|
||||
├── 统计收集器
|
||||
│ └── DatasetProfiler # 核心分析类
|
||||
│ ├── process_image() # 处理单张图片
|
||||
│ └── summarize() # 汇总统计结果
|
||||
│
|
||||
├── 可视化(9 类图表)
|
||||
│ ├── plot_overview() # 01: 概览
|
||||
│ ├── plot_2d_bbox_analysis() # 02: 2D 框分析
|
||||
│ ├── plot_3d_depth_analysis() # 03: 深度分析
|
||||
│ ├── plot_3d_dimensions() # 04: 3D 尺寸
|
||||
│ ├── plot_rotation_analysis() # 05: 旋转角
|
||||
│ ├── plot_face_visibility() # 06: 面可见性
|
||||
│ ├── plot_lateral_depth_scatter() # 07: 鸟瞰图
|
||||
│ ├── plot_depth_vs_box_size() # 08: 深度-面积
|
||||
│ └── plot_train_vs_val_comparison() # 09: Train vs Val
|
||||
│
|
||||
├── 报告生成
|
||||
│ └── generate_text_report() # 生成文本报告
|
||||
│
|
||||
└── 主流程
|
||||
├── run_profiling() # 单 split 分析流程
|
||||
└── main() # 入口(参数解析 + 调度)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 依赖
|
||||
|
||||
- Python 3.8+
|
||||
- NumPy
|
||||
- Matplotlib
|
||||
- PyYAML
|
||||
|
||||
均为 YOLOv5-3D 项目已有依赖,无需额外安装。
|
||||
1685
tools/scripts_for_gt/profiling/calib_profiling.py
Executable file
1685
tools/scripts_for_gt/profiling/calib_profiling.py
Executable file
File diff suppressed because it is too large
Load Diff
2315
tools/scripts_for_gt/profiling/dataset_profiling.py
Executable file
2315
tools/scripts_for_gt/profiling/dataset_profiling.py
Executable file
File diff suppressed because it is too large
Load Diff
77
tools/scripts_for_gt/profiling/run_calib_profiling.sh
Executable file
77
tools/scripts_for_gt/profiling/run_calib_profiling.sh
Executable file
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Calibration Parameter Profiling Launch Script
|
||||
#
|
||||
# Usage:
|
||||
# bash run_calib_profiling.sh [train|val|both]
|
||||
#
|
||||
# Positional arg (optional): split — defaults to "train"
|
||||
#
|
||||
# To also profile test-case directories and compare with training data, set
|
||||
# CASES_DIRS below (space-separated "name:path" pairs) or export the variable
|
||||
# before running this script:
|
||||
#
|
||||
# CASES_DIRS="cases_coding:/data1/dongying/Mono3d/G1M3/cases_coding cases_feishu:/data1/dongying/Mono3d/G1M3/cases_feishu" \
|
||||
# bash run_calib_profiling.sh train
|
||||
#
|
||||
# Outputs all plots and reports to $OUTPUT_DIR.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ── User-configurable parameters ─────────────────────────────────────────────
|
||||
|
||||
# Path to dataset YAML config
|
||||
DATA="data/mono3d.yaml"
|
||||
|
||||
# Which split(s) to profile: train | val | both
|
||||
SPLIT="${1:-train}"
|
||||
|
||||
# Max image files to scan (0 = all — calib is deduplicated per sequence dir)
|
||||
MAX_FILES=0
|
||||
|
||||
# Output directory for plots and reports
|
||||
OUTPUT_DIR="calib_profiling_results"
|
||||
|
||||
# Number of parallel workers (0 = auto-detect CPU count, 1 = sequential)
|
||||
WORKERS=0
|
||||
|
||||
# Test-case directories to profile alongside the training/val split.
|
||||
# All directories are merged into a single combined profile named $CASES_NAME.
|
||||
# Format: space-separated "name:path" pairs. Set to "" to skip.
|
||||
# Example:
|
||||
# CASES_DIRS="cases_coding:/data1/dongying/Mono3d/G1M3/cases_coding cases_feishu:/data1/dongying/Mono3d/G1M3/cases_feishu"
|
||||
CASES_DIRS="${CASES_DIRS:-cases_coding:/data1/dongying/Mono3d/G1M3/cases_coding cases_feishu:/data1/dongying/Mono3d/G1M3/cases_feishu}"
|
||||
|
||||
# Label used for the merged test-cases profile in plots and output filenames
|
||||
CASES_NAME="${CASES_NAME:-cases}"
|
||||
|
||||
# ── Derived paths ─────────────────────────────────────────────────────────────
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
echo "=========================================="
|
||||
echo " Calibration Profiling"
|
||||
echo " data : $DATA"
|
||||
echo " split : $SPLIT"
|
||||
echo " cases_dirs: $CASES_DIRS"
|
||||
echo " cases_name: $CASES_NAME"
|
||||
echo " output : $OUTPUT_DIR"
|
||||
echo "=========================================="
|
||||
|
||||
# Build optional --cases-dirs / --cases-name arguments
|
||||
CASES_ARGS=()
|
||||
if [[ -n "${CASES_DIRS}" ]]; then
|
||||
# shellcheck disable=SC2206
|
||||
CASES_ARGS=(--cases-dirs ${CASES_DIRS} --cases-name "${CASES_NAME}")
|
||||
fi
|
||||
|
||||
python tools/scripts_for_gt/profiling/calib_profiling.py \
|
||||
--data "$DATA" \
|
||||
--split "$SPLIT" \
|
||||
--max-files "$MAX_FILES" \
|
||||
--output-dir "$OUTPUT_DIR" \
|
||||
--workers "$WORKERS" \
|
||||
"${CASES_ARGS[@]}"
|
||||
62
tools/scripts_for_gt/profiling/run_profiling.sh
Executable file
62
tools/scripts_for_gt/profiling/run_profiling.sh
Executable file
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/env bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Dataset Profiling Launch Script
|
||||
# Usage: bash run_profiling.sh [original|roi]
|
||||
# Positional arg (optional): analysis mode — "original" (default) or "roi"
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# ── User-configurable parameters ─────────────────────────────────────────────
|
||||
|
||||
# Path to dataset YAML config
|
||||
DATA="data/mono3d.yaml"
|
||||
|
||||
# Which split(s) to profile: train | val | both
|
||||
SPLIT="train"
|
||||
|
||||
# Max files to process per split (0 = all)
|
||||
MAX_FILES=0
|
||||
|
||||
# Output directory (analysis mode suffix is appended automatically)
|
||||
OUTPUT_DIR="dataset_profiling_cncap"
|
||||
|
||||
# Image size for pixel-unit conversion [width height]
|
||||
# Leave empty ("") to use the default from YAML (ori_img_size or roi_size)
|
||||
IMG_SIZE=""
|
||||
|
||||
# Analysis mode: "original" (full image) or "roi" (crop + calibration-based filtering)
|
||||
# Can also be overridden by the first positional argument: bash run_profiling.sh roi
|
||||
ANALYSIS_MODE="${1:-original}"
|
||||
|
||||
# Number of parallel workers (0 = auto-detect CPU count, 1 = sequential)
|
||||
WORKERS=32
|
||||
|
||||
# ── Build command ─────────────────────────────────────────────────────────────
|
||||
|
||||
CMD="python tools/scripts_for_gt/profiling/dataset_profiling.py \
|
||||
--data ${DATA} \
|
||||
--split ${SPLIT} \
|
||||
--max-files ${MAX_FILES} \
|
||||
--output-dir ${OUTPUT_DIR} \
|
||||
--analysis-mode ${ANALYSIS_MODE} \
|
||||
--workers ${WORKERS}"
|
||||
|
||||
if [ -n "${IMG_SIZE}" ]; then
|
||||
CMD="${CMD} --img-size ${IMG_SIZE}"
|
||||
fi
|
||||
|
||||
# ── Run ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
cd "$(dirname "$0")/../../.." || exit 1 # ensure cwd is project root
|
||||
|
||||
echo "========================================================================"
|
||||
echo " Dataset Profiling"
|
||||
echo " data: ${DATA}"
|
||||
echo " split: ${SPLIT}"
|
||||
echo " analysis-mode: ${ANALYSIS_MODE}"
|
||||
echo " max-files: ${MAX_FILES} (0 = all)"
|
||||
echo " workers: ${WORKERS}"
|
||||
echo " output-dir: ${OUTPUT_DIR}_${ANALYSIS_MODE}"
|
||||
[ -n "${IMG_SIZE}" ] && echo " img-size: ${IMG_SIZE}"
|
||||
echo "========================================================================"
|
||||
|
||||
eval "${CMD}"
|
||||
3
tools/scripts_for_gt/profiling/run_visualize_calib.sh
Executable file
3
tools/scripts_for_gt/profiling/run_visualize_calib.sh
Executable file
@@ -0,0 +1,3 @@
|
||||
python tools/scripts_for_gt/profiling/visualize_calib_group_comparison.py \
|
||||
--input calib_profiling_results/train_calib_table.csv \
|
||||
--output-dir calib_profiling_results/group_comparison_v2
|
||||
714
tools/scripts_for_gt/profiling/visualize_calib_group_comparison.py
Executable file
714
tools/scripts_for_gt/profiling/visualize_calib_group_comparison.py
Executable file
@@ -0,0 +1,714 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Visualize calibration drift across grouped splits such as train / case / clip.
|
||||
|
||||
The script is designed for split-based calibration summary tables like
|
||||
`calib_profiling_results/train_calib_table.csv`, where each row belongs to a
|
||||
group in the `split` column and contains calibration parameters as numeric
|
||||
columns.
|
||||
|
||||
Outputs:
|
||||
1. `split_parameter_panels.png`
|
||||
Per-parameter strip plots with the baseline group's mean ± std overlay.
|
||||
2. `group_zscore_heatmap.png`
|
||||
Mean parameter drift of each non-baseline group relative to the baseline.
|
||||
3. `group_drift_ranking.png`
|
||||
Ranked horizontal bar charts of the strongest drifts per group.
|
||||
4. `parameter_pair_relationships.png`
|
||||
Selected 2D parameter relationships colored by split.
|
||||
5. `group_comparison_report.txt`
|
||||
Compact text summary with counts and top drifts.
|
||||
|
||||
Usage:
|
||||
python3 tools/scripts_for_gt/profiling/visualize_calib_group_comparison.py \
|
||||
--input calib_profiling_results/train_calib_table.csv \
|
||||
--output-dir calib_profiling_results/group_comparison
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import math
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Dict, Iterable, List, Sequence
|
||||
|
||||
try:
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
except ImportError as exc:
|
||||
raise SystemExit(
|
||||
"matplotlib is required for visualization. "
|
||||
"Install the project plotting dependencies first, for example:\n"
|
||||
" pip install -r requirements.txt"
|
||||
) from exc
|
||||
|
||||
try:
|
||||
import numpy as np
|
||||
except ImportError as exc:
|
||||
raise SystemExit(
|
||||
"numpy is required for visualization. "
|
||||
"Install the project dependencies first, for example:\n"
|
||||
" pip install -r requirements.txt"
|
||||
) from exc
|
||||
|
||||
|
||||
DEFAULT_GROUP_ORDER = ("train", "case", "clip")
|
||||
DEFAULT_PAIR_SPECS = (
|
||||
("focal_u", "focal_v"),
|
||||
("cu", "cv"),
|
||||
("yaw", "vanish_x"),
|
||||
("pitch", "vanish_y"),
|
||||
("pos_x", "pos_z"),
|
||||
("roll", "pos_z"),
|
||||
)
|
||||
PARAMETER_GROUPS = {
|
||||
"Intrinsics": ("focal_u", "focal_v", "cu", "cv"),
|
||||
"Angles": ("pitch", "yaw", "roll"),
|
||||
"Position": ("pos_x", "pos_y", "pos_z"),
|
||||
"Vanishing Point": ("vanish_x", "vanish_y"),
|
||||
}
|
||||
PALETTE = {
|
||||
"train": "#1f77b4",
|
||||
"case": "#d62728",
|
||||
"clip": "#2ca02c",
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Visualize calibration drift across grouped splits."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
type=str,
|
||||
default="calib_profiling_results/train_calib_table.csv",
|
||||
help="Input CSV with split-based calibration rows.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=str,
|
||||
default="calib_profiling_results/group_comparison",
|
||||
help="Directory where plots and report will be written.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--split-column",
|
||||
type=str,
|
||||
default="split",
|
||||
help="Column containing group labels.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--id-column",
|
||||
type=str,
|
||||
default="vehicle/date",
|
||||
help="Column containing row identifiers for optional annotations.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--baseline-split",
|
||||
type=str,
|
||||
default="train",
|
||||
help="Baseline split used for drift comparison.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--top-k",
|
||||
type=int,
|
||||
default=8,
|
||||
help="How many parameters to keep in each drift ranking panel.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--figure-dpi",
|
||||
type=int,
|
||||
default=180,
|
||||
help="Saved figure DPI.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--annotate-pairs",
|
||||
action="store_true",
|
||||
help="Annotate pair plots with row identifiers when the table is small.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _to_float(value: str) -> float:
|
||||
if value is None:
|
||||
return math.nan
|
||||
text = str(value).strip()
|
||||
if not text:
|
||||
return math.nan
|
||||
try:
|
||||
return float(text)
|
||||
except ValueError:
|
||||
return math.nan
|
||||
|
||||
|
||||
def _mean(values: Sequence[float]) -> float:
|
||||
if not values:
|
||||
return math.nan
|
||||
return float(sum(values) / len(values))
|
||||
|
||||
|
||||
def _std(values: Sequence[float]) -> float:
|
||||
if not values:
|
||||
return math.nan
|
||||
mu = _mean(values)
|
||||
return float(math.sqrt(sum((v - mu) ** 2 for v in values) / len(values)))
|
||||
|
||||
|
||||
def ordered_splits(available: Iterable[str], baseline_split: str) -> List[str]:
|
||||
available_unique = list(dict.fromkeys(available))
|
||||
ordered = [s for s in DEFAULT_GROUP_ORDER if s in available_unique]
|
||||
if baseline_split in available_unique and baseline_split not in ordered:
|
||||
ordered.insert(0, baseline_split)
|
||||
ordered.extend(s for s in available_unique if s not in ordered)
|
||||
return ordered
|
||||
|
||||
|
||||
def ordered_parameters(columns: Sequence[str]) -> List[str]:
|
||||
seen = set()
|
||||
ordered: List[str] = []
|
||||
for group_cols in PARAMETER_GROUPS.values():
|
||||
for col in group_cols:
|
||||
if col in columns and col not in seen:
|
||||
ordered.append(col)
|
||||
seen.add(col)
|
||||
for col in columns:
|
||||
if col not in seen:
|
||||
ordered.append(col)
|
||||
seen.add(col)
|
||||
return ordered
|
||||
|
||||
|
||||
def load_table(
|
||||
csv_path: Path, split_col: str, id_col: str
|
||||
) -> tuple[List[dict], List[str], List[str]]:
|
||||
with csv_path.open("r", newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
raw_rows = list(reader)
|
||||
if not raw_rows:
|
||||
raise ValueError(f"No rows found in {csv_path}")
|
||||
|
||||
columns = list(raw_rows[0].keys())
|
||||
for required in (split_col, id_col):
|
||||
if required not in columns:
|
||||
raise KeyError(
|
||||
f"Required column '{required}' not found in {csv_path}. "
|
||||
f"Available columns: {columns}"
|
||||
)
|
||||
|
||||
candidate_numeric = [c for c in columns if c not in (split_col, id_col)]
|
||||
numeric_cols = []
|
||||
for col in candidate_numeric:
|
||||
if any(not math.isnan(_to_float(row.get(col, ""))) for row in raw_rows):
|
||||
numeric_cols.append(col)
|
||||
|
||||
parsed_rows: List[dict] = []
|
||||
for row in raw_rows:
|
||||
parsed = {
|
||||
split_col: row.get(split_col, "").strip(),
|
||||
id_col: row.get(id_col, "").strip(),
|
||||
}
|
||||
for col in numeric_cols:
|
||||
parsed[col] = _to_float(row.get(col, ""))
|
||||
parsed_rows.append(parsed)
|
||||
|
||||
splits = ordered_splits(
|
||||
[row[split_col] for row in parsed_rows if row[split_col]], baseline_split=""
|
||||
)
|
||||
return parsed_rows, splits, ordered_parameters(numeric_cols)
|
||||
|
||||
|
||||
def build_group_values(
|
||||
rows: Sequence[dict], splits: Sequence[str], split_col: str, params: Sequence[str]
|
||||
) -> Dict[str, Dict[str, List[float]]]:
|
||||
grouped: Dict[str, Dict[str, List[float]]] = {
|
||||
split: {param: [] for param in params} for split in splits
|
||||
}
|
||||
for row in rows:
|
||||
split = row.get(split_col, "")
|
||||
if split not in grouped:
|
||||
continue
|
||||
for param in params:
|
||||
value = row.get(param, math.nan)
|
||||
if not math.isnan(value):
|
||||
grouped[split][param].append(value)
|
||||
return grouped
|
||||
|
||||
|
||||
def compute_baseline_stats(
|
||||
grouped_values: Dict[str, Dict[str, List[float]]],
|
||||
baseline_split: str,
|
||||
params: Sequence[str],
|
||||
) -> Dict[str, dict]:
|
||||
baseline = grouped_values.get(baseline_split)
|
||||
if baseline is None:
|
||||
raise KeyError(
|
||||
f"Baseline split '{baseline_split}' not found in input data. "
|
||||
f"Available splits: {list(grouped_values.keys())}"
|
||||
)
|
||||
|
||||
stats = {}
|
||||
for param in params:
|
||||
values = baseline[param]
|
||||
stats[param] = {
|
||||
"count": len(values),
|
||||
"mean": _mean(values),
|
||||
"std": _std(values),
|
||||
"min": min(values) if values else math.nan,
|
||||
"max": max(values) if values else math.nan,
|
||||
}
|
||||
return stats
|
||||
|
||||
|
||||
def compute_group_means(
|
||||
grouped_values: Dict[str, Dict[str, List[float]]], params: Sequence[str]
|
||||
) -> Dict[str, Dict[str, float]]:
|
||||
means: Dict[str, Dict[str, float]] = {}
|
||||
for split, param_map in grouped_values.items():
|
||||
means[split] = {param: _mean(param_map[param]) for param in params}
|
||||
return means
|
||||
|
||||
|
||||
def compute_group_zscores(
|
||||
group_means: Dict[str, Dict[str, float]],
|
||||
baseline_stats: Dict[str, dict],
|
||||
splits: Sequence[str],
|
||||
params: Sequence[str],
|
||||
) -> Dict[str, Dict[str, float]]:
|
||||
zscores: Dict[str, Dict[str, float]] = {}
|
||||
for split in splits:
|
||||
zscores[split] = {}
|
||||
for param in params:
|
||||
baseline_mean = baseline_stats[param]["mean"]
|
||||
baseline_std = baseline_stats[param]["std"]
|
||||
group_mean = group_means[split][param]
|
||||
if math.isnan(group_mean) or math.isnan(baseline_mean):
|
||||
zscores[split][param] = math.nan
|
||||
elif baseline_std == 0 or math.isnan(baseline_std):
|
||||
zscores[split][param] = 0.0 if group_mean == baseline_mean else math.nan
|
||||
else:
|
||||
zscores[split][param] = (group_mean - baseline_mean) / baseline_std
|
||||
return zscores
|
||||
|
||||
|
||||
def format_value(value: float) -> str:
|
||||
return "nan" if math.isnan(value) else f"{value:.3f}"
|
||||
|
||||
|
||||
def short_label(text: str, max_len: int = 28) -> str:
|
||||
if len(text) <= max_len:
|
||||
return text
|
||||
return text[: max_len - 3] + "..."
|
||||
|
||||
|
||||
def plot_split_parameter_panels(
|
||||
rows: Sequence[dict],
|
||||
splits: Sequence[str],
|
||||
params: Sequence[str],
|
||||
split_col: str,
|
||||
baseline_stats: Dict[str, dict],
|
||||
output_path: Path,
|
||||
dpi: int,
|
||||
) -> None:
|
||||
n_params = len(params)
|
||||
ncols = 4
|
||||
nrows = math.ceil(n_params / ncols)
|
||||
fig, axes = plt.subplots(
|
||||
nrows,
|
||||
ncols,
|
||||
figsize=(4.3 * ncols, 3.4 * nrows),
|
||||
constrained_layout=True,
|
||||
)
|
||||
axes = np.atleast_1d(axes).reshape(nrows, ncols)
|
||||
rng = np.random.default_rng(20260318)
|
||||
|
||||
for idx, param in enumerate(params):
|
||||
ax = axes[idx // ncols][idx % ncols]
|
||||
train_mean = baseline_stats[param]["mean"]
|
||||
train_std = baseline_stats[param]["std"]
|
||||
|
||||
if not math.isnan(train_mean):
|
||||
ax.axhline(train_mean, color="#444444", linewidth=1.2, linestyle="--")
|
||||
if not math.isnan(train_std) and train_std > 0:
|
||||
ax.axhspan(
|
||||
train_mean - train_std,
|
||||
train_mean + train_std,
|
||||
color="#444444",
|
||||
alpha=0.10,
|
||||
)
|
||||
|
||||
for x_idx, split in enumerate(splits):
|
||||
values = [
|
||||
row[param]
|
||||
for row in rows
|
||||
if row.get(split_col) == split and not math.isnan(row.get(param, math.nan))
|
||||
]
|
||||
if not values:
|
||||
continue
|
||||
x = x_idx + rng.uniform(-0.12, 0.12, size=len(values))
|
||||
color = PALETTE.get(split, f"C{x_idx}")
|
||||
ax.scatter(
|
||||
x,
|
||||
values,
|
||||
s=42,
|
||||
alpha=0.85,
|
||||
linewidth=0.5,
|
||||
edgecolors="white",
|
||||
color=color,
|
||||
zorder=3,
|
||||
)
|
||||
ax.scatter(
|
||||
[x_idx],
|
||||
[_mean(values)],
|
||||
marker="D",
|
||||
s=52,
|
||||
color=color,
|
||||
edgecolors="black",
|
||||
linewidth=0.7,
|
||||
zorder=4,
|
||||
)
|
||||
|
||||
ax.set_title(param, fontsize=11, pad=8)
|
||||
ax.set_xticks(range(len(splits)))
|
||||
ax.set_xticklabels(splits, rotation=20)
|
||||
ax.grid(axis="y", linestyle=":", alpha=0.35)
|
||||
ax.set_axisbelow(True)
|
||||
|
||||
for idx in range(n_params, nrows * ncols):
|
||||
axes[idx // ncols][idx % ncols].axis("off")
|
||||
|
||||
fig.suptitle(
|
||||
"Calibration Parameter Comparison by Split\n"
|
||||
"Dashed line and shaded band show baseline mean ± std",
|
||||
fontsize=14,
|
||||
)
|
||||
fig.savefig(output_path, dpi=dpi, bbox_inches="tight")
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def plot_group_zscore_heatmap(
|
||||
splits: Sequence[str],
|
||||
params: Sequence[str],
|
||||
zscores: Dict[str, Dict[str, float]],
|
||||
baseline_split: str,
|
||||
output_path: Path,
|
||||
dpi: int,
|
||||
) -> None:
|
||||
target_splits = [split for split in splits if split != baseline_split]
|
||||
if not target_splits:
|
||||
return
|
||||
|
||||
matrix = np.array(
|
||||
[[zscores[split][param] for param in params] for split in target_splits],
|
||||
dtype=float,
|
||||
)
|
||||
finite_vals = matrix[np.isfinite(matrix)]
|
||||
vmax = max(1.0, float(np.max(np.abs(finite_vals)))) if finite_vals.size else 1.0
|
||||
|
||||
fig_w = max(10.5, 0.9 * len(params))
|
||||
fig_h = max(3.6, 1.3 * len(target_splits))
|
||||
fig, ax = plt.subplots(figsize=(fig_w, fig_h), constrained_layout=True)
|
||||
|
||||
im = ax.imshow(matrix, cmap="coolwarm", vmin=-vmax, vmax=vmax, aspect="auto")
|
||||
ax.set_xticks(np.arange(len(params)))
|
||||
ax.set_xticklabels(params, rotation=35, ha="right")
|
||||
ax.set_yticks(np.arange(len(target_splits)))
|
||||
ax.set_yticklabels(target_splits)
|
||||
ax.set_title(
|
||||
f"Mean Parameter Drift vs {baseline_split} baseline (z-score units)", fontsize=13
|
||||
)
|
||||
|
||||
for i in range(len(target_splits)):
|
||||
for j in range(len(params)):
|
||||
value = matrix[i, j]
|
||||
text = "nan" if not np.isfinite(value) else f"{value:.2f}"
|
||||
color = "white" if np.isfinite(value) and abs(value) > vmax * 0.55 else "black"
|
||||
ax.text(j, i, text, ha="center", va="center", fontsize=8, color=color)
|
||||
|
||||
cbar = fig.colorbar(im, ax=ax, shrink=0.9)
|
||||
cbar.set_label("z-score of group mean relative to baseline", rotation=90)
|
||||
fig.savefig(output_path, dpi=dpi, bbox_inches="tight")
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def plot_group_drift_ranking(
|
||||
splits: Sequence[str],
|
||||
params: Sequence[str],
|
||||
zscores: Dict[str, Dict[str, float]],
|
||||
baseline_split: str,
|
||||
output_path: Path,
|
||||
dpi: int,
|
||||
top_k: int,
|
||||
) -> None:
|
||||
target_splits = [split for split in splits if split != baseline_split]
|
||||
if not target_splits:
|
||||
return
|
||||
|
||||
fig, axes = plt.subplots(
|
||||
len(target_splits),
|
||||
1,
|
||||
figsize=(10.5, 3.1 * len(target_splits)),
|
||||
constrained_layout=True,
|
||||
)
|
||||
axes = np.atleast_1d(axes)
|
||||
|
||||
for ax, split in zip(axes, target_splits):
|
||||
scored = []
|
||||
for param in params:
|
||||
value = zscores[split][param]
|
||||
if math.isnan(value):
|
||||
continue
|
||||
scored.append((param, value))
|
||||
scored.sort(key=lambda item: abs(item[1]), reverse=True)
|
||||
scored = scored[:top_k]
|
||||
|
||||
if not scored:
|
||||
ax.axis("off")
|
||||
continue
|
||||
|
||||
labels = [param for param, _ in scored][::-1]
|
||||
values = [value for _, value in scored][::-1]
|
||||
colors = ["#d62728" if value > 0 else "#1f77b4" for value in values]
|
||||
|
||||
ax.barh(labels, values, color=colors, alpha=0.88)
|
||||
ax.axvline(0.0, color="#333333", linewidth=1.0)
|
||||
ax.grid(axis="x", linestyle=":", alpha=0.35)
|
||||
ax.set_title(f"Top parameter drift for split: {split}", fontsize=12)
|
||||
ax.set_xlabel("Mean drift relative to baseline (z-score units)")
|
||||
|
||||
for y_idx, value in enumerate(values):
|
||||
text_x = value + 0.05 if value >= 0 else value - 0.05
|
||||
align = "left" if value >= 0 else "right"
|
||||
ax.text(text_x, y_idx, f"{value:.2f}", va="center", ha=align, fontsize=9)
|
||||
|
||||
fig.suptitle("Ranked group drift relative to baseline", fontsize=14)
|
||||
fig.savefig(output_path, dpi=dpi, bbox_inches="tight")
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def plot_parameter_pair_relationships(
|
||||
rows: Sequence[dict],
|
||||
splits: Sequence[str],
|
||||
split_col: str,
|
||||
id_col: str,
|
||||
output_path: Path,
|
||||
dpi: int,
|
||||
annotate_pairs: bool,
|
||||
) -> None:
|
||||
available_pairs = []
|
||||
for x_col, y_col in DEFAULT_PAIR_SPECS:
|
||||
if all(x_col in row and y_col in row for row in rows):
|
||||
if any(not math.isnan(row.get(x_col, math.nan)) for row in rows) and any(
|
||||
not math.isnan(row.get(y_col, math.nan)) for row in rows
|
||||
):
|
||||
available_pairs.append((x_col, y_col))
|
||||
|
||||
if not available_pairs:
|
||||
return
|
||||
|
||||
ncols = 2
|
||||
nrows = math.ceil(len(available_pairs) / ncols)
|
||||
fig, axes = plt.subplots(
|
||||
nrows,
|
||||
ncols,
|
||||
figsize=(6.0 * ncols, 4.8 * nrows),
|
||||
constrained_layout=True,
|
||||
)
|
||||
axes = np.atleast_1d(axes).reshape(nrows, ncols)
|
||||
|
||||
should_annotate = annotate_pairs or len(rows) <= 24
|
||||
|
||||
for idx, (x_col, y_col) in enumerate(available_pairs):
|
||||
ax = axes[idx // ncols][idx % ncols]
|
||||
for split_idx, split in enumerate(splits):
|
||||
split_rows = [
|
||||
row
|
||||
for row in rows
|
||||
if row.get(split_col) == split
|
||||
and not math.isnan(row.get(x_col, math.nan))
|
||||
and not math.isnan(row.get(y_col, math.nan))
|
||||
]
|
||||
if not split_rows:
|
||||
continue
|
||||
|
||||
xs = [row[x_col] for row in split_rows]
|
||||
ys = [row[y_col] for row in split_rows]
|
||||
ax.scatter(
|
||||
xs,
|
||||
ys,
|
||||
s=54,
|
||||
alpha=0.88,
|
||||
linewidth=0.6,
|
||||
edgecolors="white",
|
||||
color=PALETTE.get(split, f"C{split_idx}"),
|
||||
label=split,
|
||||
)
|
||||
|
||||
if should_annotate:
|
||||
for row in split_rows:
|
||||
label = short_label(row.get(id_col, ""), max_len=18)
|
||||
ax.annotate(
|
||||
label,
|
||||
(row[x_col], row[y_col]),
|
||||
textcoords="offset points",
|
||||
xytext=(4, 4),
|
||||
fontsize=7,
|
||||
alpha=0.80,
|
||||
)
|
||||
|
||||
ax.set_title(f"{x_col} vs {y_col}", fontsize=11)
|
||||
ax.set_xlabel(x_col)
|
||||
ax.set_ylabel(y_col)
|
||||
ax.grid(linestyle=":", alpha=0.35)
|
||||
|
||||
for idx in range(len(available_pairs), nrows * ncols):
|
||||
axes[idx // ncols][idx % ncols].axis("off")
|
||||
|
||||
handles, labels = axes[0][0].get_legend_handles_labels()
|
||||
if handles:
|
||||
fig.legend(handles, labels, loc="upper center", ncol=min(len(labels), 4), frameon=False)
|
||||
|
||||
fig.suptitle("Selected parameter relationships by split", fontsize=14)
|
||||
fig.savefig(output_path, dpi=dpi, bbox_inches="tight")
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def write_report(
|
||||
output_path: Path,
|
||||
csv_path: Path,
|
||||
splits: Sequence[str],
|
||||
rows: Sequence[dict],
|
||||
split_col: str,
|
||||
params: Sequence[str],
|
||||
baseline_split: str,
|
||||
grouped_values: Dict[str, Dict[str, List[float]]],
|
||||
baseline_stats: Dict[str, dict],
|
||||
group_means: Dict[str, Dict[str, float]],
|
||||
zscores: Dict[str, Dict[str, float]],
|
||||
top_k: int,
|
||||
) -> None:
|
||||
counts = defaultdict(int)
|
||||
for row in rows:
|
||||
counts[row.get(split_col, "")] += 1
|
||||
|
||||
lines = []
|
||||
lines.append("=" * 72)
|
||||
lines.append("CALIBRATION GROUP COMPARISON REPORT")
|
||||
lines.append("=" * 72)
|
||||
lines.append(f"Input CSV : {csv_path}")
|
||||
lines.append(f"Baseline split : {baseline_split}")
|
||||
lines.append("")
|
||||
lines.append("Split counts:")
|
||||
for split in splits:
|
||||
lines.append(f" {split:<24} {counts.get(split, 0)}")
|
||||
lines.append("")
|
||||
lines.append("Baseline summary:")
|
||||
for param in params:
|
||||
stat = baseline_stats[param]
|
||||
lines.append(
|
||||
f" {param:<24} n={stat['count']:<3d} "
|
||||
f"mean={format_value(stat['mean'])} std={format_value(stat['std'])} "
|
||||
f"range=[{format_value(stat['min'])}, {format_value(stat['max'])}]"
|
||||
)
|
||||
|
||||
for split in splits:
|
||||
if split == baseline_split:
|
||||
continue
|
||||
lines.append("")
|
||||
lines.append(f"Top drift vs baseline for split '{split}':")
|
||||
ranked = []
|
||||
for param in params:
|
||||
z_value = zscores[split][param]
|
||||
group_mean = group_means[split][param]
|
||||
base_mean = baseline_stats[param]["mean"]
|
||||
if math.isnan(z_value) or math.isnan(group_mean) or math.isnan(base_mean):
|
||||
continue
|
||||
ranked.append((param, z_value, group_mean - base_mean))
|
||||
ranked.sort(key=lambda item: abs(item[1]), reverse=True)
|
||||
for param, z_value, delta in ranked[:top_k]:
|
||||
lines.append(
|
||||
f" {param:<24} delta={delta:+.3f} z={z_value:+.2f}"
|
||||
)
|
||||
|
||||
output_path.write_text("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
csv_path = Path(args.input)
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
rows, file_splits, params = load_table(
|
||||
csv_path=csv_path,
|
||||
split_col=args.split_column,
|
||||
id_col=args.id_column,
|
||||
)
|
||||
splits = ordered_splits(file_splits, args.baseline_split)
|
||||
grouped_values = build_group_values(rows, splits, args.split_column, params)
|
||||
baseline_stats = compute_baseline_stats(grouped_values, args.baseline_split, params)
|
||||
group_means = compute_group_means(grouped_values, params)
|
||||
zscores = compute_group_zscores(group_means, baseline_stats, splits, params)
|
||||
|
||||
plot_split_parameter_panels(
|
||||
rows=rows,
|
||||
splits=splits,
|
||||
params=params,
|
||||
split_col=args.split_column,
|
||||
baseline_stats=baseline_stats,
|
||||
output_path=output_dir / "split_parameter_panels.png",
|
||||
dpi=args.figure_dpi,
|
||||
)
|
||||
plot_group_zscore_heatmap(
|
||||
splits=splits,
|
||||
params=params,
|
||||
zscores=zscores,
|
||||
baseline_split=args.baseline_split,
|
||||
output_path=output_dir / "group_zscore_heatmap.png",
|
||||
dpi=args.figure_dpi,
|
||||
)
|
||||
plot_group_drift_ranking(
|
||||
splits=splits,
|
||||
params=params,
|
||||
zscores=zscores,
|
||||
baseline_split=args.baseline_split,
|
||||
output_path=output_dir / "group_drift_ranking.png",
|
||||
dpi=args.figure_dpi,
|
||||
top_k=args.top_k,
|
||||
)
|
||||
plot_parameter_pair_relationships(
|
||||
rows=rows,
|
||||
splits=splits,
|
||||
split_col=args.split_column,
|
||||
id_col=args.id_column,
|
||||
output_path=output_dir / "parameter_pair_relationships.png",
|
||||
dpi=args.figure_dpi,
|
||||
annotate_pairs=args.annotate_pairs,
|
||||
)
|
||||
write_report(
|
||||
output_path=output_dir / "group_comparison_report.txt",
|
||||
csv_path=csv_path,
|
||||
splits=splits,
|
||||
rows=rows,
|
||||
split_col=args.split_column,
|
||||
params=params,
|
||||
baseline_split=args.baseline_split,
|
||||
grouped_values=grouped_values,
|
||||
baseline_stats=baseline_stats,
|
||||
group_means=group_means,
|
||||
zscores=zscores,
|
||||
top_k=args.top_k,
|
||||
)
|
||||
|
||||
print("Saved outputs to:", output_dir)
|
||||
print(" - split_parameter_panels.png")
|
||||
print(" - group_zscore_heatmap.png")
|
||||
print(" - group_drift_ranking.png")
|
||||
print(" - parameter_pair_relationships.png")
|
||||
print(" - group_comparison_report.txt")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user