zhao.zhu/yolov26_3d
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

单目3D初始代码

Browse Source
This commit is contained in:
zhao.zhu
2026-06-24 09:35:46 +08:00
commit 04a5895b6b
1153 changed files with 340700 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

363
tools/convert_gt_to_label/convert_json_to_json.md Executable file
View File

@@ -0,0 +1,363 @@
# `convert_json_to_json.py` 说明文档
## 1. 脚本用途
`tools/convert_gt_to_label/convert_json_to_json.py` 的作用是：
- 读取原始 GT 标注 JSON；
- 从 `asso_list` 中提取每个目标的 `camera_mea` 和可选的 `lidar_mea`；
- 结合标定文件，将原始标注转换为项目当前使用的目标 JSON 格式；
- 在输出中同时写入：
- 2D 框信息；
- 相机系 3D 信息；
- ego 系 3D 信息；
- 车辆前后左右面的附加信息（仅部分目标类型会生成）。
这个脚本本质上是一个“原始标注格式 -> 最终训练/评估使用 JSON 格式”的转换器，而不是简单的字段重命名脚本。
---
## 2. 输入与输出
### 2.1 单文件模式
命令格式：
```bash
python3 tools/convert_gt_to_label/convert_json_to_json.py \
input.json \
output.json \
--calib-path /path/to/case/calib
```
说明：
- `input.json`：原始标注文件。
- `output.json`：转换后的输出文件。
- `--calib-path`：标定目录。脚本会读取：
```text
<calib-path>/L2_calib/camera4.json
```
如果不传 `--calib-path`，代码默认使用 `input.json` 的父目录，但只有当该目录下确实存在 `L2_calib/camera4.json` 时才可正常工作。
### 2.2 批量模式
命令格式：
```bash
python3 tools/convert_gt_to_label/convert_json_to_json.py \
--json-root /path/to/dataset_root
```
或使用封装脚本：
```bash
bash tools/convert_gt_to_label/run_convert_json_to_json.sh /path/to/dataset_root
```
批量模式下，脚本会递归查找：
```text
<json-root>/**/annotations_20260320/*.json
```
并假设每个 case 目录结构类似：
```text
case_xxx/
├── annotations_20260320/
│ └── xxx.json
├── calib/
│ └── L2_calib/camera4.json
└── labels_20260320_with_ego/
└── xxx.json # 脚本输出
```
注意：代码当前实际输出目录是 `labels_20260320_with_ego`。
---
## 3. 转换主流程
单个目标的大致处理链路如下：
1. 从原始标注的 `asso_list` 中逐个读取对象。
2. 从 `camera_mea` 读取：
- `cls`
- 2D 框位置
- `confidence`
3. 用 `DETECTION_CLASS_MAP` 将原始类别名映射为项目内部类别 id。
4. 若目标存在 `lidar_mea`，则进一步计算：
- 相机系 3D 中心点、尺寸、朝向；
- ego 系 3D 信息；
- 部分目标的前/后/左/右面信息。
5. 将最终结果组织成目标 JSON 条目，包含：
- `type`
- `type_name`
- `box2d`
- `3d_ori`
- `3d_front/back/left/right`
- `3d_ori_ego`
- `3d_ori_ego_from_cam`
- 一致性校验字段等。
---
## 4. `DETECTION_CLASS_MAP` 的作用
`DETECTION_CLASS_MAP` 是这个脚本里最关键的类别标准化入口。
代码位置：
- 定义：`tools/convert_gt_to_label/convert_json_to_json.py`
- 使用：`build_label_row()` 中先取 `raw_label_name`，再执行 `label_det_cls = DETECTION_CLASS_MAP.get(raw_label_name, -1)`
它的职责不是单纯“把字符串换成数字”，而是同时决定了下面三件事：
### 4.1 决定原始类别如何映射到输出类别
当前代码会先取原始类别名：
- 若 `camera_mea["cls"] == "vehicle"`，则使用 `camera_mea["subcls"]`
- 否则直接使用 `camera_mea["cls"]`
随后脚本会先通过 `DETECTION_CLASS_MAP` 把原始类别归一化成标准类别 id，再通过 `CLASS_MAP_MODEL` 得到标准类别名，最终写入输出中的：
- `type`：类别 id
- `type_name`：标准类别名
例如：
- `tanker -> 6 -> truck`
- `large_truck -> 6 -> truck`
- `motorcyclist -> 10 -> bicyclist`
- `tricyclist -> 12 -> tricycle`
这说明 `DETECTION_CLASS_MAP` 做的是“原始类别标准化”，不是“原样透传”。
### 4.2 决定目标是否被直接过滤
如果映射结果为 `-1`，目标会被直接丢弃，不写入输出。
当前脚本中：
- `carton -> -1`
对应逻辑在 `build_label_row()` 中：
- 若 `label_det_cls < 0`，函数直接返回 `None`
- 上层 `generate_label_rows()` 会跳过该目标
也就是说，`DETECTION_CLASS_MAP` 同时承担了“保留/过滤白名单”的作用。
### 4.3 间接决定是否生成 3D 信息
在 `build_3d_label()` 中，存在如下规则：
- 若 `label_lidar is None`，不生成有效 3D；
- 若类别 id 不在 `SUPPORTED_3D_CLASS_IDS` 中，不生成有效 3D；
- 若类别 id 在 `COMPLETE_3D_CLASS_IDS` 中，只生成 `3d_ori`，不生成 `3d_front/back/left/right`；
因此，`DETECTION_CLASS_MAP` 还会影响 3D 处理分支。
结合当前类别定义，可理解为：
- `0~8`：`face_3d_classes`，可进入 3D 处理，且会额外尝试生成 `3d_front/back/left/right`
- `9~12`：`complete_3d_classes`，可进入 3D 处理，但只保留 `3d_ori`
- `13~16`：只保留 2D 或空 3D 占位
- `-1`：直接过滤，不输出
所以，`DETECTION_CLASS_MAP` 不只是“类别映射表”，它还在事实上定义了哪些类别属于当前 3D 支持范围。
---
## 5. 当前 `DETECTION_CLASS_MAP` 解释
当前映射如下：
| 原始类别 / 子类 | 映射后 id | 输出标准类 | 3D 行为 |
| --- | ---: | --- |
| `car` | 0 | `car` | `face_3d` |
| `suv` | 1 | `suv` | `face_3d` |
| `pickup` | 2 | `pickup` | `face_3d` |
| `medium_car` | 3 | `medium_car` | `face_3d` |
| `van` | 4 | `van` | `face_3d` |
| `bus` | 5 | `bus` | `face_3d` |
| `truck` / `tanker` / `large_truck` / `construction_vehicle` | 6 | `truck` | `face_3d` |
| `special_vehicle` | 7 | `special_vehicle` | `face_3d` |
| `unknown` | 8 | `unknown` | `face_3d` |
| `pedestrian` | 9 | `pedestrian` | `complete_3d` |
| `bicyclist` / `motorcyclist` | 10 | `bicyclist` | `complete_3d` |
| `bicycle` / `motorcycle` | 11 | `bicycle` | `complete_3d` |
| `tricycle` / `tricyclist` | 12 | `tricycle` | `complete_3d` |
| `traffic_sign` | 13 | `traffic_sign` | 仅 2D |
| `wheel` | 14 | `wheel` | 仅 2D |
| `plate` | 15 | `plate` | 仅 2D |
| `face` | 16 | `face` | 仅 2D |
可以看出，这张表体现了三类策略：
### 5.1 一对一映射
例如：
- `pedestrian -> 9`
- `wheel -> 14`
适用于原始类别与项目标准类别完全一致的情况。
### 5.2 多对一归并
例如：
- `tanker`
- `large_truck`
- `construction_vehicle`
都被归并为 `truck(6)`。
这意味着训练/评估侧不再区分这些细粒度障碍物，而是把它们当成同一大类处理。
### 5.3 显式过滤
例如：
- `carton -> -1`
说明该类别虽然可能出现在原始标注中，但当前转换目标并不希望保留它。
---
## 6. `DETECTION_CLASS_MAP` 的维护原则
后续如果原始 GT 中新增了 `camera_mea.cls` 类别，最先需要检查的就是这张表。
### 6.1 如果原始类别未出现在映射表中
当前代码会直接执行：
```python
label_det_cls = DETECTION_CLASS_MAP.get(raw_label_name, -1)
```
这意味着：
- 未知类别会被映射为 `-1`；
- `build_label_row()` 会直接返回 `None`，该目标不会写入输出。
因此，只要数据源里出现新类别，就必须同步更新 `DETECTION_CLASS_MAP`，否则会被静默过滤。
### 6.2 建议先用统计脚本盘点类别
同目录已经提供：
```text
tools/convert_gt_to_label/stat_gt_categories.py
```
它可以统计原始 GT 中所有 `camera_mea.cls` 的取值，并生成映射模板，适合在更新 `DETECTION_CLASS_MAP` 前先做盘点。
建议流程：
1. 先运行 `stat_gt_categories.py` 统计现网类别；
2. 确认每个原始类别应该：
- 映射到哪个标准类；
- 还是直接过滤；
3. 再更新 `DETECTION_CLASS_MAP`；
4. 最后运行 `convert_json_to_json.py` 做验证。
### 6.3 修改映射时要同步关注 3D 语义
例如把某个新类别映射到：
- `0~8`：表示该类别会进入 `face_3d` 分支；
- `9~12`：表示该类别会进入 `complete_3d` 分支；
- `13~16`：表示该类别只保留 2D 或空 3D 占位；
- `-1`：表示完全过滤。
所以改这张表时，不能只从“类别名是否合理”来判断，还要同时确认：
- 是否希望输出 3D；
- 是否希望参与车辆面信息计算；
- 是否会影响下游训练/评估类别定义。
---
## 7. 输出类别与 `CLASS_MAP` 的关系
脚本中还有一张表：
```python
CLASS_MAP = {
"car": 0,
"suv": 1,
"pickup": 2,
...
}
```
它和 `DETECTION_CLASS_MAP` 的关系可以理解为：
- `DETECTION_CLASS_MAP`：原始类别名 / 原始 `subcls` -> 内部标准 id
- `CLASS_MAP_MODEL`：内部标准 id -> 规范化后的标准类别名
- `CLASS_MAP`：标准类别名 -> 输出类别 id
两者配合后，最终输出对象里会写入：
- `type`: 例如 `6`
- `type_name`: 例如 `truck`
---
## 8. 使用这个脚本时最容易踩的点
### 8.1 类别未配置会被直接过滤
如果原始类别或车辆 `subcls` 出现新值，但 `DETECTION_CLASS_MAP` 没更新，该目标会被跳过，不会写入输出。
### 8.2 批量模式只扫描固定目录名
批量模式只查找：
```text
annotations/*.json
```
如果目录名变化，需要修改 `find_annotation_files()`。
### 8.3 批量模式输出目录是 `labels_json`
批量模式当前实际写入的是：
```text
labels_json
```
使用时应以代码实际行为为准。
### 8.4 并非所有类别都会生成有效 3D
即使原始数据里存在 `lidar_mea`，也只有：
- `0~8` 类会生成 `face_3d`
- `9~12` 类会生成 `complete_3d`
- `13~16` 类会写成空 3D
### 8.5 输出类别名是归一化后的标准类名
例如：
- `tanker` 最终输出为 `truck`
- `motorcyclist` 最终输出为 `bicyclist`
- `tricyclist` 最终输出为 `tricycle`
---
## 9. 一句话总结
如果只抓一个重点来理解这个脚本，那么就是：
`DETECTION_CLASS_MAP` 是原始 GT 到项目标准标签体系的总入口，它同时决定了“类别如何归一化成标准输出、目标是否过滤、以及目标进入哪种 3D 分支”。
理解这张表，基本就理解了这个脚本一半以上的业务语义。

1408
tools/convert_gt_to_label/convert_json_to_json.py Executable file
View File

File diff suppressed because it is too large Load Diff

1397
tools/convert_gt_to_label/convert_json_to_txt.py Executable file
View File

File diff suppressed because it is too large Load Diff

233
tools/convert_gt_to_label/convert_results_to_example_format.py Executable file
View File

@@ -0,0 +1,233 @@
#!/usr/bin/env python3
# coding: utf-8
import argparse
import json
from pathlib import Path
from typing import Any
from uuid import NAMESPACE_URL, uuid5
DEFAULT_RESULTS_DIR = "/data1/dongying/Mono3d/D4Q2/feishu_project/0416_chenyile/results"
DEFAULT_EXAMPLE_JSON = "/data1/dongying/Mono3d/D4Q2/feishu_project/0416_dongying/example_format.json"
DEFAULT_VIS_DIR = "/data1/dongying/Mono3d/D4Q2/feishu_project/0416_chenyile/vis"
DEFAULT_OUTPUT_JSON = "/data1/dongying/Mono3d/D4Q2/feishu_project/0416_chenyile/results_example_format.json"
DEFAULT_USER_ID = 1698
DEFAULT_SOURCE = "default"
DEFAULT_IMAGE_SIZE = [0, 0]
IMAGE_SUFFIXES = (".jpg", ".jpeg", ".png", ".bmp", ".webp")
LABEL_OVERRIDES = {
"pedestrian": (["pedestrian", "pedestrian"], ["行人", "行人"]),
"face": (["sensitive_area", "face"], ["敏感区域", "人脸"]),
"suv": (["vehicle", "suv"], ["机动车", "SUV"]),
"car": (["vehicle", "car"], ["机动车", "小轿车"]),
"truck": (["vehicle", "truck"], ["机动车", "卡车"]),
"traffic_sign": (["traffic_sign", "traffic_sign_unknown"], ["交通标志", "未知交通标志"]),
"plate": (["sensitive_area", "plate"], ["敏感区域", "车牌"]),
"wheel": (["vehicle", "wheel"], ["机动车", "车轮"]),
"traffic_cone": (["road_barrier", "traffic_cone"], ["路障", "交通锥"]),
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="将结果目录重排为 example_format.json 同款结构。")
parser.add_argument("results_dir", nargs="?", default=DEFAULT_RESULTS_DIR, help="输入结果目录。")
parser.add_argument("output_json", nargs="?", default=DEFAULT_OUTPUT_JSON, help="输出 JSON 文件。")
parser.add_argument("--example-json", default=DEFAULT_EXAMPLE_JSON, help="示例 JSON，用于补充已知标签映射。")
parser.add_argument("--vis-dir", default=DEFAULT_VIS_DIR, help="可选图片目录，用于补 image_name 和 image_size。")
parser.add_argument("--userid", type=int, default=DEFAULT_USER_ID, help="输出 annotations.userid。")
parser.add_argument("--source", default=DEFAULT_SOURCE, help="输出 annotations.source。")
parser.add_argument(
"--unknown-occlusion",
default="no",
choices=["no", "slight", "partial", "heavy"],
help="结果里 unknown_occlusion 的回填值。",
)
return parser.parse_args()
def load_example_label_map(example_json: Path) -> dict[str, tuple[list[str], list[str]]]:
if not example_json.is_file():
return {}
with example_json.open("r", encoding="utf-8") as file_obj:
records = json.load(file_obj)
label_map: dict[str, tuple[list[str], list[str]]] = {}
for record in records:
for annotation_block in record.get("annotations", []):
for annotation in annotation_block.get("anno", []):
label = annotation.get("label") or []
label_desc = annotation.get("labelDesc") or []
if len(label) < 2 or len(label_desc) < 2:
continue
child_label = str(label[1])
label_map[child_label] = ([str(item) for item in label], [str(item) for item in label_desc])
return label_map
def resolve_label_mapping(name: str, example_label_map: dict[str, tuple[list[str], list[str]]]) -> tuple[list[str], list[str]]:
if name in LABEL_OVERRIDES:
return LABEL_OVERRIDES[name]
if name in example_label_map:
return example_label_map[name]
return (["vehicle", "unknown"], ["机动车", "未知车辆"])
def normalize_occlusion(raw_occlusion: Any, unknown_occlusion: str) -> str:
mapping = {
"no_occlusion": "no",
"slight_occlusion": "slight",
"partial_occlusion": "partial",
"heavy_occlusion": "heavy",
"unknown_occlusion": unknown_occlusion,
None: "no",
}
return mapping.get(raw_occlusion, unknown_occlusion)
def stable_uuid(prefix: str, value: str) -> str:
return str(uuid5(NAMESPACE_URL, f"{prefix}:{value}"))
def build_vis_image_map(vis_dir: Path) -> dict[str, Path]:
if not vis_dir.is_dir():
raise FileNotFoundError(f"vis directory not found: {vis_dir}")
vis_image_map: dict[str, Path] = {}
for image_path in sorted(vis_dir.rglob("*")):
if not image_path.is_file() or image_path.suffix.lower() not in IMAGE_SUFFIXES:
continue
vis_image_map.setdefault(image_path.stem, image_path)
return vis_image_map
def detect_image_info(stem: str, vis_image_map: dict[str, Path]) -> tuple[str, list[int]]:
image_path = vis_image_map.get(stem)
if image_path is None:
raise FileNotFoundError(f"vis image not found for result frame: {stem}")
return image_path.name, list(DEFAULT_IMAGE_SIZE)
def convert_detection(
detection: dict[str, Any],
stem: str,
index: int,
example_label_map: dict[str, tuple[list[str], list[str]]],
unknown_occlusion: str,
) -> dict[str, Any]:
bbox_xyxy = detection.get("bbox_xyxy")
if not isinstance(bbox_xyxy, list) or len(bbox_xyxy) != 4:
raise ValueError(f"invalid bbox_xyxy in {stem}: {detection}")
label, label_desc = resolve_label_mapping(str(detection.get("name", "unknown")), example_label_map)
fake_cls_id = detection.get("fake_cls_id", -1)
annotation_attr = {
"is_fake": "true" if fake_cls_id not in (-1, None) else "false",
"is_inferred": "false",
"occlusion": normalize_occlusion(detection.get("occlusion"), unknown_occlusion),
"truncation": "no",
}
return {
"attr": annotation_attr,
"data": {
"point2d": [
[float(bbox_xyxy[0]), float(bbox_xyxy[1])],
[float(bbox_xyxy[2]), float(bbox_xyxy[3])],
]
},
"id": index,
"index": stable_uuid("annotation-index", f"{stem}:{index}"),
"label": label,
"labelDesc": label_desc,
"type": "rect",
}
def convert_result_file(
result_path: Path,
example_label_map: dict[str, tuple[list[str], list[str]]],
vis_image_map: dict[str, Path],
userid: int,
source: str,
unknown_occlusion: str,
) -> dict[str, Any]:
with result_path.open("r", encoding="utf-8") as file_obj:
detections = json.load(file_obj)
if not isinstance(detections, list):
raise ValueError(f"result file must be a list: {result_path}")
stem = result_path.stem
image_name, image_size = detect_image_info(stem, vis_image_map)
anno = [
convert_detection(detection, stem, index, example_label_map, unknown_occlusion)
for index, detection in enumerate(detections)
]
return {
"annotations": [
{
"anno": anno,
"anno_uuid": stable_uuid("anno", stem),
"source": source,
"userid": userid,
}
],
"image_attr": None,
"image_name": image_name,
"image_size": image_size,
"image_uuid": stable_uuid("image", stem),
"package_image_uuid": stable_uuid("package-image", stem),
}
def main() -> int:
args = parse_args()
results_dir = Path(args.results_dir).resolve()
output_json = Path(args.output_json).resolve()
example_json = Path(args.example_json).resolve()
vis_dir = Path(args.vis_dir).resolve()
if not results_dir.is_dir():
raise FileNotFoundError(f"results directory not found: {results_dir}")
example_label_map = load_example_label_map(example_json)
vis_image_map = build_vis_image_map(vis_dir)
result_files = sorted(results_dir.glob("*.json"))
if not result_files:
raise ValueError(f"no json files found under {results_dir}")
selected_result_files = [result_path for result_path in result_files if result_path.stem in vis_image_map]
skipped_result_files = [result_path for result_path in result_files if result_path.stem not in vis_image_map]
if not selected_result_files:
raise ValueError(f"no result frames matched vis images under {vis_dir}")
converted = [
convert_result_file(
result_path,
example_label_map,
vis_image_map,
userid=args.userid,
source=args.source,
unknown_occlusion=args.unknown_occlusion,
)
for result_path in selected_result_files
]
output_json.parent.mkdir(parents=True, exist_ok=True)
with output_json.open("w", encoding="utf-8") as file_obj:
json.dump(converted, file_obj, ensure_ascii=False, indent=2)
print(
f"result_files={len(result_files)} matched_vis_files={len(selected_result_files)} "
f"skipped_files={len(skipped_result_files)} output={output_json}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

1243
tools/convert_gt_to_label/converter.py Executable file
View File

File diff suppressed because it is too large Load Diff

277
tools/convert_gt_to_label/count_visualization_batches.py Executable file
View File

@@ -0,0 +1,277 @@
#!/usr/bin/env python3
# coding: utf-8
import argparse
import json
import os
from datetime import datetime
from pathlib import Path
DEFAULT_SHOWPATH_ROOT = "/data1/dongying/Mono3d/D4Q2/data_visualization_for_check_camera2"
DEFAULT_BATCH_GLOB = "batch_*"
DEFAULT_IMAGE_EXTS = ".jpg,.jpeg,.png"
def parse_args():
parser = argparse.ArgumentParser(
description="统计可视化结果根目录下每个 batch 的 2D/3D 图像数量。"
)
parser.add_argument(
"--showpath-root",
default=DEFAULT_SHOWPATH_ROOT,
help="可视化结果根目录，目录下应包含 batch_* 子目录。",
)
parser.add_argument(
"--batch-glob",
default=DEFAULT_BATCH_GLOB,
help="batch 目录匹配模式，默认 batch_*。",
)
parser.add_argument(
"--image-exts",
default=DEFAULT_IMAGE_EXTS,
help="参与统计的图片扩展名，逗号分隔，默认 .jpg,.jpeg,.png。",
)
parser.add_argument(
"--check-pairs",
action="store_true",
help="额外校验 2D/3D 下的相对图片路径是否一一对应。",
)
parser.add_argument(
"--sample-mismatch-limit",
type=int,
default=10,
help="配对校验失败时，最多展示多少条 only_in_2d/only_in_3d 样例。",
)
parser.add_argument(
"--output-json",
default=None,
help="可选，将统计结果写入 JSON 文件。",
)
return parser.parse_args()
def normalize_image_exts(raw_exts):
image_exts = []
for ext in raw_exts.split(","):
ext = ext.strip().lower()
if not ext:
continue
if not ext.startswith("."):
ext = f".{ext}"
image_exts.append(ext)
if not image_exts:
raise ValueError("image-exts 不能为空。")
return tuple(sorted(set(image_exts)))
def collect_image_stats(image_dir, image_exts, collect_relpaths=False):
count = 0
relpaths = set() if collect_relpaths else None
if not image_dir.is_dir():
return count, relpaths
for current_root, _, filenames in os.walk(image_dir):
current_root_path = Path(current_root)
for filename in filenames:
if Path(filename).suffix.lower() not in image_exts:
continue
count += 1
if relpaths is not None:
file_path = current_root_path / filename
relpaths.add(file_path.relative_to(image_dir).as_posix())
return count, relpaths
def build_status(batch_summary, check_pairs):
issues = []
if not batch_summary["has_2d_dir"]:
issues.append("MISSING_2D")
if not batch_summary["has_3d_dir"]:
issues.append("MISSING_3D")
if batch_summary["count_diff"] != 0:
issues.append("COUNT_DIFF")
if check_pairs and (
batch_summary["only_in_2d_count"] > 0 or batch_summary["only_in_3d_count"] > 0
):
issues.append("PAIR_DIFF")
return "OK" if not issues else "+".join(issues)
def analyze_batch(batch_dir, image_exts, check_pairs=False, sample_mismatch_limit=10):
dir_2d = batch_dir / "2D"
dir_3d = batch_dir / "3D"
count_2d, relpaths_2d = collect_image_stats(
dir_2d, image_exts, collect_relpaths=check_pairs
)
count_3d, relpaths_3d = collect_image_stats(
dir_3d, image_exts, collect_relpaths=check_pairs
)
summary = {
"batch_name": batch_dir.name,
"batch_dir": str(batch_dir),
"has_2d_dir": dir_2d.is_dir(),
"has_3d_dir": dir_3d.is_dir(),
"count_2d": count_2d,
"count_3d": count_3d,
"count_diff": count_2d - count_3d,
}
if check_pairs:
only_in_2d = sorted(relpaths_2d - relpaths_3d)
only_in_3d = sorted(relpaths_3d - relpaths_2d)
summary.update(
{
"only_in_2d_count": len(only_in_2d),
"only_in_3d_count": len(only_in_3d),
"only_in_2d_samples": only_in_2d[:sample_mismatch_limit],
"only_in_3d_samples": only_in_3d[:sample_mismatch_limit],
}
)
summary["status"] = build_status(summary, check_pairs)
return summary
def build_total_row(batch_summaries, check_pairs):
total_row = {
"batch_name": "TOTAL",
"count_2d": sum(item["count_2d"] for item in batch_summaries),
"count_3d": sum(item["count_3d"] for item in batch_summaries),
}
total_row["count_diff"] = total_row["count_2d"] - total_row["count_3d"]
if check_pairs:
total_row["only_in_2d_count"] = sum(
item["only_in_2d_count"] for item in batch_summaries
)
total_row["only_in_3d_count"] = sum(
item["only_in_3d_count"] for item in batch_summaries
)
total_row["status"] = "OK"
if any(item["status"] != "OK" for item in batch_summaries):
total_row["status"] = "HAS_ISSUES"
return total_row
def print_summary_table(batch_summaries, total_row, check_pairs):
columns = [
("batch_name", "batch"),
("count_2d", "2D"),
("count_3d", "3D"),
("count_diff", "diff"),
]
if check_pairs:
columns.extend(
[
("only_in_2d_count", "only_2d"),
("only_in_3d_count", "only_3d"),
]
)
columns.append(("status", "status"))
table_rows = batch_summaries + [total_row]
widths = {}
for key, title in columns:
widths[key] = max(
len(title),
max(len(str(row.get(key, ""))) for row in table_rows),
)
header = " ".join(title.ljust(widths[key]) for key, title in columns)
separator = " ".join("-" * widths[key] for key, _ in columns)
print(header)
print(separator)
for row in table_rows:
print(
" ".join(str(row.get(key, "")).ljust(widths[key]) for key, _ in columns)
)
def build_report(args, batch_summaries, total_row, image_exts):
return {
"generated_at": datetime.now().isoformat(timespec="seconds"),
"showpath_root": str(Path(args.showpath_root).resolve()),
"batch_glob": args.batch_glob,
"image_exts": list(image_exts),
"check_pairs": args.check_pairs,
"sample_mismatch_limit": args.sample_mismatch_limit,
"total_batches": len(batch_summaries),
"total_summary": total_row,
"batches": batch_summaries,
}
def print_pair_mismatch_details(batch_summaries):
mismatched_batches = [
item
for item in batch_summaries
if item["only_in_2d_count"] > 0 or item["only_in_3d_count"] > 0
]
if not mismatched_batches:
return
print("\nPair mismatch details:")
for item in mismatched_batches:
print(
f"- {item['batch_name']}: only_in_2d={item['only_in_2d_count']}, "
f"only_in_3d={item['only_in_3d_count']}"
)
if item["only_in_2d_samples"]:
print(f" only_in_2d samples: {item['only_in_2d_samples']}")
if item["only_in_3d_samples"]:
print(f" only_in_3d samples: {item['only_in_3d_samples']}")
def main():
args = parse_args()
image_exts = normalize_image_exts(args.image_exts)
showpath_root = Path(args.showpath_root)
if not showpath_root.is_dir():
print(f"可视化结果根目录不存在: {showpath_root}")
return 1
batch_dirs = sorted(
path for path in showpath_root.glob(args.batch_glob) if path.is_dir()
)
if not batch_dirs:
print(
f"未找到 batch 目录，root={showpath_root}, batch_glob={args.batch_glob}"
)
return 1
batch_summaries = [
analyze_batch(
batch_dir,
image_exts,
check_pairs=args.check_pairs,
sample_mismatch_limit=args.sample_mismatch_limit,
)
for batch_dir in batch_dirs
]
total_row = build_total_row(batch_summaries, args.check_pairs)
print_summary_table(batch_summaries, total_row, args.check_pairs)
if args.check_pairs:
print_pair_mismatch_details(batch_summaries)
report = build_report(args, batch_summaries, total_row, image_exts)
if args.output_json:
output_json = Path(args.output_json)
output_json.parent.mkdir(parents=True, exist_ok=True)
with open(output_json, "w", encoding="utf-8") as f:
json.dump(report, f, ensure_ascii=False, indent=2)
print(f"\n统计结果已写入: {output_json}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

1610
tools/convert_gt_to_label/example.json Executable file
View File

File diff suppressed because it is too large Load Diff

4028
tools/convert_gt_to_label/example_converted.json Executable file
View File

File diff suppressed because it is too large Load Diff

61
tools/convert_gt_to_label/example_converted.txt Executable file
View File

@@ -0,0 +1,61 @@
0 0.4478526041666667 0.5342865740740741 0.021103125000000014 0.032756481481481435 -5.322678325758643 0.9269782749746722 62.31183979627242 4.2753500939967495 1.5497414206240265 1.8667275547695945 -1.6644804990396993 0.4484375 0.5361111111111111 0.44895833333333335 0.5527777777777778 -1.5792673365929495 2 -5.5226518245574 0.9269782749746723 64.4401408103063 -1.5789873571664765 0.4484375 0.5351851851851852 0.0 1.0 -5.122704826959886 0.9269782749746723 60.18353878223853 -1.5795671329741405 0.44895833333333335 0.5370370370370371 0.9972042575955438 1.0 -6.251949159817303 0.9269782749746722 62.22452623545889 -1.5643425280922993 0.4375 0.5361111111111111 0.0 -1.0 -4.393407491699982 0.9269782749746723 62.39915335708595 -1.5941883724040449 0.45989583333333334 0.5361111111111111 0.0 -1.0 0.96
0 0.4192578125 0.5268273148148148 0.016055208333333345 0.028360185185185204 -11.178087914511165 1.126623659839888 91.10809174378281 4.3394085453321996 1.638317293102368 1.8966120615775184 -1.6809316457528798 0.42135416666666664 0.5324074074074074 0.42135416666666664 0.5444444444444444 -1.5588513648773974 2 -11.416566188320822 1.126623659839888 93.26465028547517 -1.5591271864689538 0.42135416666666664 0.5324074074074074 0.0 1.0 -10.939609640701509 1.126623659839888 88.95153320209045 -1.5585621891865657 0.42083333333333334 0.5333333333333333 0.9960981407977217 1.0 -12.120648380746003 1.126623659839888 91.00386077179513 -1.5485226283513929 0.41354166666666664 0.5324074074074074 0.0 -1.0 -10.235527448276326 1.126623659839888 91.21232271577048 -1.569182650706946 0.42864583333333334 0.5324074074074074 0.03205149144398053 -1.0 0.96
0 0.4901458333333333 0.5398986111111111 0.03416041666666665 0.056117592592592566 -1.2207727474111165 0.7417515610941243 40.95386615446082 4.255067102022341 1.602117779670393 1.9317511054345955 -1.6031243112161284 0.490625 0.5407407407407407 0.490625 0.5675925925925925 -1.573324649211065 2 -1.2895396394130736 0.7417515610941243 43.08028806114882 -1.5731998432485674 0.490625 0.5398148148148149 0.0 1.0 -1.1520058554091595 0.7417515610941243 38.827444247772824 -1.5734631264946095 0.490625 0.5416666666666666 0.9991507706959047 1.0 -2.1861436265007925 0.7417515610941243 40.92264678318958 -1.5497536786784258 0.47291666666666665 0.5407407407407407 0.0 -1.0 -0.25540186832144046 0.7417515610941243 40.98508552573206 -1.5968928111680094 0.5083333333333333 0.5407407407407407 0.0 -1.0 0.95
0 0.5539872395833334 0.5398157407407407 0.04805468749999999 0.06785370370370374 1.7965333771789131 0.6810791278357566 35.80506950417485 4.364611388811167 1.6452826101350673 1.9515451783566093 -1.6303546516568754 0.5515625 0.5416666666666666 0.5515625 0.5731481481481482 -1.6804879998811413 2 1.6666357328349335 0.6810791278357566 37.98350581167363 -1.674204405842794 0.546875 0.5407407407407407 0.0 1.0 1.9264310215228928 0.6810791278357566 33.626633196676075 -1.6875809468832172 0.5567708333333333 0.5435185185185185 0.9621084147554989 1.0 0.8224909038548001 0.6810791278357566 35.746988474979624 -1.6533592694725814 0.5307291666666667 0.5416666666666666 0.22926229906695217 -1.0 2.7705758505030262 0.6810791278357566 35.86315053337008 -1.7074556224660682 0.571875 0.5416666666666666 0.0 -1.0 0.94
0 0.43622265624999995 0.5246351851851851 0.007328645833333347 0.01318333333333328 -1 0.94
0 0.4685973958333333 0.527699074074074 0.007571875000000006 0.018261111111111093 -1 0.93
0 0.52715078125 0.5331833333333333 0.02214843750000005 0.04639814814814816 1.2888118094408063 0.7312509380193262 52.36979611561394 4.493534510282726 1.6635234267603494 1.8789202854762639 -1.647023413387698 0.5322916666666667 0.5351851851851852 0.5322916666666667 0.5564814814814815 -1.6716282774998268 2 1.1177130965291244 0.7312509380193262 54.61003903307075 -1.6674877288067362 0.5291666666666667 0.5342592592592592 0.0 1.0 1.4599105223524882 0.7312509380193262 50.129553198157126 -1.6761379357716946 0.5354166666666667 0.5361111111111111 0.9658868716740873 1.0 0.35207974420975546 0.7312509380193262 52.29825313719172 -1.6537554628208084 0.5182291666666666 0.5351851851851852 0.22981171922386145 -1.0 2.225543874671857 0.7312509380193262 52.44133909403615 -1.6894366936903278 0.5458333333333333 0.5351851851851852 0.0 -1.0 0.92
0 0.10260625 0.49172916666666666 0.06731458333333332 0.0626805555555555 -28.223506943782933 -0.5737357863269261 32.63411845801956 11.245817250335156 3.34837425993108 2.9292775402711784 1.508036394208803 0.09895833333333333 0.5009259259259259 0.09947916666666666 0.5444444444444444 2.221086787579271 0 -27.870845195179363 -0.5737357863269259 27.022279980474448 2.308891823505762 0.07083333333333333 0.5 0.7280891517516188 1.0 -28.576168692386503 -0.5737357863269259 38.245956935564664 2.149722785033046 0.12552083333333333 0.5018518518518519 0.0 1.0 -26.761751693277454 -0.5737357863269259 32.725978757409216 2.1935086988816606 0.10885416666666667 0.5009259259259259 0.6296276745572366 1.0 -29.685262194288413 -0.5737357863269259 32.54225815862989 2.2475546793311034 0.09010416666666667 0.5018518518518519 0.0 1.0 0.92
0 0.47313958333333334 0.5319606481481481 0.00781874999999997 0.026975000000000034 -3.346141769945419 1.1405364422438105 73.97580981833369 4.276902474722242 1.6040241866108331 1.8584808008962146 -1.668303332348193 0.4786458333333333 0.5370370370370371 0.4791666666666667 0.5518518518518518 -1.6231012252264567 2 -3.5543254918500575 1.1405364422438105 76.10410332076826 -1.6216337775450165 0.47760416666666666 0.5361111111111111 0.0 1.0 -3.1379580480407796 1.1405364422438105 71.84751631589911 -1.6246558238118147 0.48020833333333335 0.537962962962963 0.982706456307913 1.0 -4.270968238709756 1.1405364422438105 73.88534587801576 -1.610562238832687 0.46927083333333336 0.5370370370370371 0.15910928907354044 -1.0 -2.421315301181082 1.1405364422438105 74.06627375865162 -1.6356237716506252 0.48854166666666665 0.5370370370370371 0.0 -1.0 0.91
0 0.2927854166666667 0.5616333333333334 0.09309479166666668 0.11907222222222229 -5.688442457002822 0.6515146009446231 18.847581783882784 4.572232225876627 1.6314661906678796 2.032986999088073 -1.6067899425915653 0.30364583333333334 0.5592592592592592 0.3046875 0.612037037037037 -1.3136708098830607 2 -5.770710275806388 0.6515146009446231 21.132217178825595 -1.3402129439854304 0.32083333333333336 0.5546296296296296 0.0 1.0 -5.6061746381992545 0.6515146009446231 16.562946388939974 -1.2804173492789355 0.2833333333333333 0.5638888888888889 0.9028504867548437 1.0 -6.704277573445244 0.6515146009446231 18.81100240693766 -1.2644232676909533 0.2734375 0.5574074074074075 0.0 -1.0 -4.6726073405604 0.6515146009446231 18.884161160827908 -1.3642266045932105 0.33645833333333336 0.5601851851851852 0.3626380696039398 -1.0 0.91
7 0.5116114583333333 0.46365925925925927 0.025406249999999984 0.028024074074074038 -1 0.91
0 0.46437369791666666 0.5231402777777777 0.008083854166666645 0.018084259259259213 -1 0.91
0 0.3746828125 0.5191787037037038 0.011252083333333355 0.010085185185185233 -1 0.89
0 0.42897421874999997 0.5274851851851852 0.00588385416666668 0.017529629629629645 -1 0.89
9 0.6749369791666666 0.717326388888889 0.01799479166666664 0.11860462962962959 -1 0.88
8 0.7758239583333334 0.6332930555555556 0.03819895833333339 0.030591666666666656 -1 0.88
9 0.6255473958333333 0.6657171296296297 0.009577083333333292 0.08832870370370369 -1 0.87
0 0.7247682291666667 0.6297800925925925 0.20937604166666665 0.294475 2.434552068275443 0.5967205147944805 8.058303786791164 4.350524874593896 1.6536647566476377 2.0479167613181763 -1.6151998089257185 0.7223958333333333 0.6074074074074074 0.7182291666666667 0.7287037037037037 -1.908597842744447 2 2.3379945787164735 0.5967205147944805 10.231422127407967 -1.8398537412718614 0.6776041666666667 0.5907407407407408 0.0 1.0 2.5311095578344127 0.5967205147944805 5.885185446174361 -2.0213666722443397 0.7880208333333333 0.6314814814814815 0.844486831895201 1.0 1.4116029753817916 0.5967205147944805 8.012851408745153 -1.78957793404185 0.6427083333333333 0.6138888888888889 0.3732032917911752 -1.0 3.4575011611690947 0.5967205147944805 8.103756164837176 -2.018470697720395 0.7875 0.6 0.0 -1.0 0.87
8 0.2723901041666667 0.5561902777777779 0.016830208333333315 0.012158333333333307 -1 0.86
8 0.49003854166666666 0.5378004629629629 0.008782291666666648 0.005245370370370337 -1 0.86
8 0.557528125 0.5356800925925925 0.010732291666666664 0.006441666666666661 -1 0.85
0 0.39026536458333333 0.5196268518518518 0.010867187500000005 0.008009259259259239 -1 0.85
9 0.3358854166666667 0.5885976851851852 0.005460416666666686 0.04338240740740736 -1 0.85
9 0.3146812499999999 0.5978162037037037 0.007009374999999984 0.04718796296296293 -1 0.84
0 0.31288854166666674 0.5064208333333334 0.029720833333333314 0.0355490740740741 -1 0.84
2 0.8897486979166668 0.571062962962963 0.03627968749999996 0.09224814814814819 8.433443163792141 0.5561401950767102 10.904898338138215 1.5817150292535551 1.1590158531086248 0.6559058475241687 2.2699693057766948 0.9072916666666667 0.562037037037037 0.9057291666666667 0.6101851851851852 1.6116828731051827 -1 0.84
0 0.15874635416666666 0.5144449074074074 0.05282708333333333 0.07640092592592596 -21.61443310326651 0.6272993643326822 33.165049334711064 4.393192427520202 1.6734817142825604 1.9250059218333837 1.5111592429980747 0.15208333333333332 0.5351851851851852 0.15260416666666668 0.5592592592592592 2.0887449199425503 0 -21.483512148192084 0.6272993643326823 30.972358150383545 2.117600620303752 0.13958333333333334 0.5351851851851852 0.8073992662473614 1.0 -21.745354058340936 0.6272993643326823 35.35774051903858 2.062542259413652 0.16354166666666667 0.5342592592592592 0.0 1.0 -20.653641245359005 0.6272993643326823 33.22241618534834 2.067366050077159 0.16145833333333334 0.5351851851851852 0.562250926367033 1.0 -22.575224961174015 0.6272993643326823 33.10768248407379 2.109615354828887 0.14322916666666666 0.5342592592592592 0.0 1.0 0.83
2 0.8415385416666666 0.5651731481481481 0.02687291666666667 0.07048703703703701 8.274189751848002 0.6023878510463183 15.041911630616367 1.5969253707069795 1.1521873497832582 0.6506352301708902 -2.3274015159349246 0.8395833333333333 0.5583333333333333 0.8390625 0.5990740740740741 -2.830302827792134 -1 0.79
2 0.9305695312500001 0.5717587962962962 0.04436510416666669 0.08631388888888883 8.57433515265391 0.499378573554066 9.216428684195302 1.613578829060235 1.1916644060725476 0.666386172630037 -2.173991236345734 0.9390625 0.5601851851851852 0.9375 0.6129629629629629 -2.923313660642581 -1 0.79
5 0.6147372395833334 0.5129231481481481 0.00301510416666666 0.0058499999999999846 -1 0.74
5 0.6297578125 0.5220236111111111 0.002812500000000047 0.005423148148148121 -1 0.71
5 0.6562223958333333 0.5084175925925926 0.0033177083333332763 0.007281481481481512 -1 0.69
5 0.6463380208333334 0.5099624999999999 0.004018749999999945 0.00903425925925921 -1 0.69
1 0.6151989583333334 0.5296060185185185 0.007279166666666725 0.039167592592592636 7.347362924447552 0.5188912729640653 53.535415266887306 0.621501340976369 1.6793547489833478 0.6376270827755769 1.392885077955592 0.615625 0.5287037037037037 0.6161458333333333 0.55 1.2564941201312294 -1 0.67
0 0.34099843750000003 0.510698611111111 0.021696875000000008 0.02038796296296297 -1 0.66
1 0.6062763020833334 0.5306763888888889 0.006914062500000048 0.04047314814814816 6.684211922727918 0.50565346419902 52.309795522222025 0.6372101484471651 1.7148569604954305 0.661297859795197 1.4078590097619292 0.6088541666666667 0.5287037037037037 0.6088541666666667 0.5509259259259259 1.2807664813456126 -1 0.66
5 0.6063796875 0.5144546296296296 0.0034864583333333123 0.0063851851851851466 -1 0.66
5 0.6237174479166666 0.5085476851851851 0.0036546875000000273 0.006847222222222205 -1 0.63
0 0.3362307291666667 0.522824074074074 0.016811458333333345 0.022809259259259272 -1 0.58
0 0.21077135416666667 0.5157 0.01811562499999999 0.0124092592592593 -1 0.57
0 0.010891927083333334 0.5274768518518518 0.015023437499999999 0.07451851851851854 -28.552226792399612 0.49877847130119835 20.052826453506107 4.421758635933981 1.635324433067478 1.9300756313973146 1.5296674075878407 0.013541666666666667 0.5314814814814814 0.0140625 0.5574074074074075 2.4881817971862157 0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 0.0 -1.0 -28.643132235093383 0.4987784713011984 22.261836086600276 2.4397720971159007 0.0359375 0.5305555555555556 1.0 1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 0.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 0.0 -1.0 0.54
0 0.19245078125 0.5186064814814815 0.015506770833333346 0.008390740740740752 -1 0.53
0 0.36203151041666665 0.5207194444444444 0.013595312499999975 0.024218518518518473 -1 0.52
1 0.6457776041666666 0.5293657407407407 0.008578125000000015 0.04807222222222223 7.264055455404972 0.4250015235932112 38.97695214074227 0.6340940521295699 1.6317781099034117 0.6495978470487854 1.520944803155322 0.65 0.5305555555555556 0.65 0.5574074074074075 1.3366906605014859 -1 0.5
8 0.4186299479166667 0.5273449074074075 0.004230729166666692 0.002800925925925905 -1 0.49
1 0.6234333333333334 0.5265527777777778 0.006395833333333319 0.04297407407407411 7.435751816404869 0.5914538128258182 51.34862219983651 0.6215427178226364 1.6906295073461521 0.6293591509621058 -1.7423619244841528 0.6213541666666667 0.5314814814814814 0.6213541666666667 0.5537037037037037 -1.8861714493903388 -1 0.48
0 0.23057760416666667 0.5147611111111111 0.030210416666666677 0.014929629629629652 -1 0.47
5 0.5779479166666667 0.5136111111111111 0.0028249999999999885 0.005500000000000051 -1 0.46
0 0.35159765625000006 0.5188657407407408 0.008054687500000017 0.00868148148148146 -1 0.45
8 0.4477528645833333 0.5327592592592592 0.005349479166666645 0.0034777777777777515 -1 0.43
1 0.6304580729166667 0.5335208333333332 0.009071354166666623 0.02864722222222219 7.435045285601043 0.6536059457746322 47.49907910714277 0.6325653300177067 1.3389352886791703 0.6131565073809939 2.999690514771298 0.6296875 0.5342592592592592 0.6296875 0.5527777777777778 2.8444201609579514 -1 0.4
5 0.8994692708333333 0.48519537037037036 0.0030354166666666533 0.007194444444444428 -1 0.4
0 0.45893307291666663 0.5263041666666667 0.003116145833333306 0.008612037037037078 -1 0.38
0 0.23074557291666664 0.5255541666666668 0.032185937499999984 0.03686203703703707 -1 0.37
0 0.03305703125 0.5166648148148149 0.027794270833333336 0.014185185185185231 -1 0.36
1 0.8995971354166666 0.4979037037037037 0.005455729166666619 0.03278888888888892 -1 0.34
0 0.4071604166666667 0.5223296296296296 0.008616666666666658 0.009635185185185137 -1 0.34
0 0.011420052083333333 0.5057291666666667 0.012668229166666666 0.02998240740740738 -1 0.32
5 0.6049046874999999 0.5127273148148148 0.0031270833333332843 0.006182407407407426 -1 0.32
0 0.35157239583333333 0.5235708333333333 0.008809374999999993 0.018156481481481516 -1 0.31
6 0.5291265625 0.5049412037037037 0.0037979166666666825 0.006150925925925953 -1 0.3

1053
tools/convert_gt_to_label/package_visualization_batches.py Executable file
View File

File diff suppressed because it is too large Load Diff

53
tools/convert_gt_to_label/package_visualization_batches.sh Executable file
View File

@@ -0,0 +1,53 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
PYTHON_BIN="${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}"
detect_cpu_count() {
local cpu_count
cpu_count="$(getconf _NPROCESSORS_ONLN 2>/dev/null || nproc 2>/dev/null || echo 1)"
if [[ ! "${cpu_count}" =~ ^[0-9]+$ ]] || (( cpu_count <= 0 )); then
cpu_count=1
fi
echo "${cpu_count}"
}
calc_scan_workers() {
local cpu_count="$1"
if (( cpu_count >= 8 )); then
echo 8
else
echo "${cpu_count}"
fi
}
calc_part_workers() {
local cpu_count="$1"
if (( cpu_count >= 32 )); then
echo 4
elif (( cpu_count >= 16 )); then
echo 3
elif (( cpu_count >= 8 )); then
echo 2
else
echo 1
fi
}
CPU_COUNT="$(detect_cpu_count)"
# 打包是 I/O + gzip 压缩混合负载，这里默认给一套偏稳妥的并发配置；
# 如需手动调节，可在命令前覆盖这些环境变量。
VIS_PACK_SCAN_WORKERS="${VIS_PACK_SCAN_WORKERS:-$(calc_scan_workers "${CPU_COUNT}")}"
VIS_PACK_ARCHIVE_WORKERS="${VIS_PACK_ARCHIVE_WORKERS:-2}"
VIS_PACK_PART_WORKERS="${VIS_PACK_PART_WORKERS:-$(calc_part_workers "${CPU_COUNT}")}"
exec "${PYTHON_BIN}" \
"${PROJECT_ROOT}/tools/convert_gt_to_label/package_visualization_batches.py" \
--scan-workers "${VIS_PACK_SCAN_WORKERS}" \
--archive-workers "${VIS_PACK_ARCHIVE_WORKERS}" \
--part-workers "${VIS_PACK_PART_WORKERS}" \
"$@"

498
tools/convert_gt_to_label/rewrite_visualization_archives.py Executable file
View File

@@ -0,0 +1,498 @@
#!/usr/bin/env python3
# coding: utf-8
import argparse
import copy
import hashlib
import json
import tarfile
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
from pathlib import Path, PurePosixPath
MODALITIES = ("2D", "3D")
DEFAULT_ARCHIVE_GLOB = "part_*.tar.gz"
DEFAULT_REPORT_NAME = "rewrite_archives_report.json"
def parse_args():
parser = argparse.ArgumentParser(
description=(
"将已有可视化归档中的成员路径改写为 part_xxxx/images/<filename>，"
"并输出到新的归档目录。"
)
)
parser.add_argument(
"source_path",
help="输入路径，可为 archive root、单个 modality 目录，或单个 tar/tar.gz 文件。",
)
parser.add_argument(
"output_root",
nargs="?",
default=None,
help="输出目录。默认在 source_path 同级生成 <name>_images_prefixed。",
)
parser.add_argument(
"--modalities",
default="2D,3D",
help="当 source_path 为 archive root 时要处理的模态，逗号分隔，默认 2D,3D。",
)
parser.add_argument(
"--archive-glob",
default=DEFAULT_ARCHIVE_GLOB,
help="归档匹配模式，默认 part_*.tar.gz。",
)
parser.add_argument(
"--parts",
default=None,
help=(
"仅处理指定 part，逗号分隔；既支持 part_0001，也支持 part_0001.tar.gz。"
),
)
parser.add_argument(
"--limit",
type=int,
default=None,
help="最多处理多少个归档，便于抽样验证。",
)
parser.add_argument(
"--workers",
type=int,
default=1,
help="并发处理归档数，默认 1。",
)
parser.add_argument(
"--gzip-compresslevel",
type=int,
default=1,
help="输出 tar.gz 的 gzip 压缩级别，范围 0-9，默认 1。",
)
parser.add_argument(
"--checksum",
action="store_true",
help="输出归档后计算 sha256。",
)
parser.add_argument(
"--verify",
action="store_true",
default=True,
help="输出归档后校验成员数及路径前缀，默认开启。",
)
parser.add_argument(
"--no-verify",
dest="verify",
action="store_false",
help="关闭输出归档校验。",
)
parser.add_argument(
"--overwrite",
action="store_true",
help="允许覆盖已有输出归档。",
)
parser.add_argument(
"--report-name",
default=DEFAULT_REPORT_NAME,
help=f"输出报告文件名，默认 {DEFAULT_REPORT_NAME}。",
)
parser.add_argument(
"--dry-run",
action="store_true",
help="仅输出转换计划，不实际生成归档。",
)
return parser.parse_args()
def now_str():
return datetime.now().isoformat(timespec="seconds")
def normalize_modalities(raw_modalities):
modalities = []
for item in raw_modalities.split(","):
modality = item.strip()
if not modality:
continue
if modality not in MODALITIES:
raise ValueError(f"unsupported modality: {modality}")
modalities.append(modality)
if not modalities:
raise ValueError("modalities 不能为空。")
return tuple(dict.fromkeys(modalities))
def archive_suffix(path):
name = path.name
if name.endswith(".tar.gz"):
return ".tar.gz"
if name.endswith(".tar"):
return ".tar"
raise ValueError(f"unsupported archive suffix: {path}")
def strip_archive_suffix(name):
if name.endswith(".tar.gz"):
return name[: -len(".tar.gz")]
if name.endswith(".tar"):
return name[: -len(".tar")]
return name
def parse_parts_filter(raw_parts):
if not raw_parts:
return None
normalized = []
for item in raw_parts.split(","):
value = item.strip()
if not value:
continue
normalized.append(strip_archive_suffix(value))
if not normalized:
return None
return set(normalized)
def is_archive_file(path):
return path.is_file() and path.name.endswith((".tar", ".tar.gz"))
def default_output_root(source_path, resolve_mode):
source_path = source_path.resolve()
if resolve_mode == "single_archive":
base_name = strip_archive_suffix(source_path.name)
else:
base_name = source_path.name
return source_path.parent / f"{base_name}_images_prefixed"
def resolve_archives(source_path, archive_glob, modalities):
source_path = source_path.resolve()
if source_path.is_file():
if not is_archive_file(source_path):
raise ValueError(f"source_path is not a supported archive file: {source_path}")
return "single_archive", [source_path], source_path.parent
if not source_path.is_dir():
raise FileNotFoundError(f"source_path does not exist or is not a directory: {source_path}")
direct_archives = sorted(path for path in source_path.glob(archive_glob) if is_archive_file(path))
if source_path.name in MODALITIES and direct_archives:
return "modality_dir", direct_archives, source_path.parent
archive_paths = []
for modality in modalities:
modality_dir = source_path / modality
if not modality_dir.is_dir():
continue
archive_paths.extend(
sorted(path for path in modality_dir.glob(archive_glob) if is_archive_file(path))
)
if archive_paths:
return "archive_root", archive_paths, source_path
raise FileNotFoundError(
f"no archives found under {source_path} with archive_glob={archive_glob}"
)
def filter_archives(archive_paths, parts_filter, limit):
selected = archive_paths
if parts_filter:
selected = [
path for path in selected if strip_archive_suffix(path.name) in parts_filter
]
if limit is not None:
selected = selected[:limit]
if not selected:
raise FileNotFoundError("没有匹配到待处理归档，请检查 parts 或 archive_glob。")
return selected
def build_output_archive_path(archive_path, source_base_dir, output_root):
return output_root / archive_path.relative_to(source_base_dir)
def build_archive_member_dir(archive_path):
return f"{strip_archive_suffix(archive_path.name)}/images"
def build_output_member_name(member_name, output_archive):
return f"{build_archive_member_dir(output_archive)}/{PurePosixPath(member_name).name}"
def open_output_tarfile(output_archive, archive_format, gzip_compresslevel):
suffix = archive_format or archive_suffix(output_archive)
if suffix == ".tar.gz":
return tarfile.open(
output_archive,
mode="w:gz",
compresslevel=gzip_compresslevel,
)
return tarfile.open(output_archive, mode="w")
def compute_sha256(file_path):
digest = hashlib.sha256()
with open(file_path, "rb") as f:
while True:
chunk = f.read(1024 * 1024)
if not chunk:
break
digest.update(chunk)
return digest.hexdigest()
def verify_rewritten_archive(archive_path, expected_member_count):
member_count = 0
samples = []
expected_prefix = f"{build_archive_member_dir(archive_path)}/"
with tarfile.open(archive_path, mode="r:*") as tar_obj:
for member in tar_obj:
if not member.isfile():
raise ValueError(
f"rewritten archive contains non-regular member: {archive_path} -> {member.name}"
)
if not member.name.startswith(expected_prefix):
raise ValueError(
f"rewritten archive member missing expected prefix {expected_prefix}: "
f"{archive_path} -> {member.name}"
)
member_count += 1
if len(samples) < 5:
samples.append(member.name)
if member_count != expected_member_count:
raise ValueError(
f"rewritten archive member count mismatch: {archive_path}, "
f"expected={expected_member_count}, actual={member_count}"
)
return {
"member_count": member_count,
"sample_members": samples,
}
def rewrite_archive(
source_archive,
output_archive,
gzip_compresslevel,
verify,
checksum,
overwrite,
dry_run,
):
source_archive = source_archive.resolve()
output_archive = output_archive.resolve()
if output_archive.exists() and not overwrite:
raise FileExistsError(f"output archive already exists: {output_archive}")
samples_before = []
samples_after = []
if dry_run:
return {
"status": "dry_run",
"source_archive": str(source_archive),
"output_archive": str(output_archive),
"member_count": None,
"sample_members_before": samples_before,
"sample_members_after": samples_after,
"output_sha256": None,
"verification": None,
}
output_archive.parent.mkdir(parents=True, exist_ok=True)
partial_archive = output_archive.with_name(f"{output_archive.name}.partial")
if partial_archive.exists():
partial_archive.unlink()
seen_output_names = {}
member_count = 0
output_archive_format = archive_suffix(output_archive)
try:
with tarfile.open(source_archive, mode="r:*") as input_tar:
with open_output_tarfile(
partial_archive,
output_archive_format,
gzip_compresslevel,
) as output_tar:
for member in input_tar:
if not member.isfile():
raise ValueError(
f"only regular file members are supported: "
f"{source_archive} -> {member.name}"
)
output_name = build_output_member_name(member.name, output_archive)
if output_name in seen_output_names:
raise ValueError(
"duplicated output member name after rewrite: "
f"{output_name}, first={seen_output_names[output_name]}, "
f"duplicate={member.name}"
)
seen_output_names[output_name] = member.name
if len(samples_before) < 5:
samples_before.append(member.name)
if len(samples_after) < 5:
samples_after.append(output_name)
input_file = input_tar.extractfile(member)
if input_file is None:
raise ValueError(
f"failed to extract file object from member: {source_archive} -> {member.name}"
)
output_member = copy.copy(member)
output_member.name = output_name
if output_member.pax_headers:
output_member.pax_headers = dict(output_member.pax_headers)
try:
output_tar.addfile(output_member, input_file)
finally:
input_file.close()
member_count += 1
partial_archive.replace(output_archive)
finally:
if partial_archive.exists():
partial_archive.unlink()
verification = verify_rewritten_archive(output_archive, member_count) if verify else None
output_sha256 = compute_sha256(output_archive) if checksum else None
return {
"status": "rewritten",
"source_archive": str(source_archive),
"output_archive": str(output_archive),
"member_count": member_count,
"sample_members_before": samples_before,
"sample_members_after": samples_after,
"output_sha256": output_sha256,
"verification": verification,
}
def rewrite_archives(archive_paths, source_base_dir, output_root, args):
tasks = [
(archive_path, build_output_archive_path(archive_path, source_base_dir, output_root))
for archive_path in archive_paths
]
if args.workers <= 1 or len(tasks) <= 1:
return [
rewrite_archive(
source_archive,
output_archive,
args.gzip_compresslevel,
args.verify,
args.checksum,
args.overwrite,
args.dry_run,
)
for source_archive, output_archive in tasks
]
results = [None] * len(tasks)
with ThreadPoolExecutor(max_workers=min(args.workers, len(tasks))) as executor:
future_map = {
executor.submit(
rewrite_archive,
source_archive,
output_archive,
args.gzip_compresslevel,
args.verify,
args.checksum,
args.overwrite,
args.dry_run,
): index
for index, (source_archive, output_archive) in enumerate(tasks)
}
for future in as_completed(future_map):
index = future_map[future]
results[index] = future.result()
return results
def build_report(args, resolve_mode, source_path, output_root, archive_paths, results):
return {
"generated_at": now_str(),
"source_path": str(Path(source_path).resolve()),
"output_root": str(Path(output_root).resolve()),
"resolve_mode": resolve_mode,
"modalities": list(normalize_modalities(args.modalities)),
"archive_glob": args.archive_glob,
"parts": sorted(parse_parts_filter(args.parts) or []),
"limit": args.limit,
"workers": args.workers,
"gzip_compresslevel": args.gzip_compresslevel,
"checksum": args.checksum,
"verify": args.verify,
"overwrite": args.overwrite,
"dry_run": args.dry_run,
"selected_archive_count": len(archive_paths),
"results": results,
}
def write_json(path, data):
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=2)
def main():
args = parse_args()
if args.workers <= 0:
raise ValueError("workers 必须大于 0")
if not 0 <= args.gzip_compresslevel <= 9:
raise ValueError("gzip-compresslevel 必须在 0 到 9 之间")
source_path = Path(args.source_path).resolve()
modalities = normalize_modalities(args.modalities)
parts_filter = parse_parts_filter(args.parts)
resolve_mode, archive_paths, source_base_dir = resolve_archives(
source_path,
args.archive_glob,
modalities,
)
archive_paths = filter_archives(archive_paths, parts_filter, args.limit)
output_root = (
Path(args.output_root).resolve()
if args.output_root
else default_output_root(source_path, resolve_mode)
)
print("")
print("######################################################################")
print("# Rewrite visualization archives")
print("######################################################################")
print(f"Source path : {source_path}")
print(f"Output root : {output_root}")
print(f"Resolve mode : {resolve_mode}")
print(f"Selected count : {len(archive_paths)}")
print(f"Workers : {args.workers}")
print(f"Dry run : {args.dry_run}")
results = rewrite_archives(archive_paths, source_base_dir, output_root, args)
report = build_report(args, resolve_mode, source_path, output_root, archive_paths, results)
report_path = output_root / args.report_name
write_json(report_path, report)
print(f"Report : {report_path}")
for item in results:
print(
f"[{item['status']}] {item['source_archive']} -> {item['output_archive']}"
)
if item["sample_members_before"]:
print(f" before: {item['sample_members_before'][:3]}")
print(f" after : {item['sample_members_after'][:3]}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

40
tools/convert_gt_to_label/rewrite_visualization_archives.sh Executable file
View File

@@ -0,0 +1,40 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
PYTHON_BIN="${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}"
detect_cpu_count() {
local cpu_count
cpu_count="$(getconf _NPROCESSORS_ONLN 2>/dev/null || nproc 2>/dev/null || echo 1)"
if [[ ! "${cpu_count}" =~ ^[0-9]+$ ]] || (( cpu_count <= 0 )); then
cpu_count=1
fi
echo "${cpu_count}"
}
calc_workers() {
local cpu_count="$1"
if (( cpu_count >= 16 )); then
echo 4
elif (( cpu_count >= 8 )); then
echo 2
else
echo 1
fi
}
CPU_COUNT="$(detect_cpu_count)"
# 归档重写主要是串行读取 + gzip 写出，默认给一套偏稳妥的并发配置；
# 如需手动调节，可在命令前覆盖这些环境变量。
VIS_REWRITE_WORKERS="${VIS_REWRITE_WORKERS:-$(calc_workers "${CPU_COUNT}")}"
VIS_REWRITE_GZIP_LEVEL="${VIS_REWRITE_GZIP_LEVEL:-1}"
exec "${PYTHON_BIN}" \
"${PROJECT_ROOT}/tools/convert_gt_to_label/rewrite_visualization_archives.py" \
--workers "${VIS_REWRITE_WORKERS}" \
--gzip-compresslevel "${VIS_REWRITE_GZIP_LEVEL}" \
"$@"

27
tools/convert_gt_to_label/run_convert_json_to_json.sh Executable file
View File

@@ -0,0 +1,27 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHON_SCRIPT="${SCRIPT_DIR}/convert_json_to_json.py"
DEFAULT_DEV_PYTHON="/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python"
PYTHON_BIN="${PYTHON_BIN:-}"
if [[ -z "${PYTHON_BIN}" ]]; then
if [[ -x "${DEFAULT_DEV_PYTHON}" ]]; then
PYTHON_BIN="${DEFAULT_DEV_PYTHON}"
else
PYTHON_BIN="python3"
fi
fi
if [[ $# -lt 1 ]]; then
echo "Usage: $0 <json_root> [extra args...]"
echo "Example: $0 /data1/dongying/Mono3d/G1M3/Testdata_0129/"
exit 1
fi
JSON_ROOT="$1"
shift
"${PYTHON_BIN}" "${PYTHON_SCRIPT}" --json-root "${JSON_ROOT}" "$@"

23
tools/convert_gt_to_label/run_convert_json_to_json_test.sh Executable file
View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
set -euo pipefail
DEFAULT_DEV_PYTHON="/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python"
PYTHON_BIN="${PYTHON_BIN:-}"
if [[ -z "${PYTHON_BIN}" ]]; then
if [[ -x "${DEFAULT_DEV_PYTHON}" ]]; then
PYTHON_BIN="${DEFAULT_DEV_PYTHON}"
else
PYTHON_BIN="python3"
fi
fi
CASE_DIR="/data1/dongying/Mono3d/G1Q3/dataset_for_evaluation/OP_KPI_SCENE/019b1a9c-6a6a-7523-b69c-748bdbd74d33"
INPUT_JSON="${CASE_DIR}/annotations/G1M3_2232_20251212195850_019b1a9c-6a6a-7523-b69c-748bdbd74d33_000001_49704_1765542681704437000.json"
OUTPUT_JSON="${CASE_DIR}/annotations_converted/G1M3_2232_20251212195850_019b1a9c-6a6a-7523-b69c-748bdbd74d33_000001_49704_1765542681704437000_converted.json"
CALIB_DIR="${CASE_DIR}/calib"
"${PYTHON_BIN}" tools/convert_gt_to_label/convert_json_to_json.py \
"${INPUT_JSON}" \
"${OUTPUT_JSON}" \
--calib-path "${CALIB_DIR}"

172
tools/convert_gt_to_label/stat_gt_categories.py Executable file
View File

@@ -0,0 +1,172 @@
#!/usr/bin/env python3
# coding: utf-8
import argparse
import glob
import json
import os
from collections import Counter
DEFAULT_JSON_ROOT = "/data1/dongying/Mono3d/G1M3/Testdata_0129"
DEFAULT_GLOB_PATTERN = os.path.join(DEFAULT_JSON_ROOT, "*", "annotations_20260320", "*.json")
def parse_args():
parser = argparse.ArgumentParser(
description="统计原始真值中的 camera_mea.cls 类别，并生成映射模板。"
)
parser.add_argument(
"--json-root",
default=DEFAULT_JSON_ROOT,
help="原始标注根目录。若未显式传 glob-pattern，则会自动拼接默认 pattern。",
)
parser.add_argument(
"--glob-pattern",
default=None,
help="JSON 搜索 pattern，优先级高于 json-root。",
)
parser.add_argument(
"--output-dir",
default=os.path.join(os.path.dirname(__file__), "category_stats_output"),
help="统计结果输出目录。",
)
parser.add_argument(
"--limit",
type=int,
default=0,
help="仅统计前 N 个 JSON，0 表示统计全部。",
)
parser.add_argument(
"--sample-per-category",
type=int,
default=5,
help="每个类别保留的样例数量。",
)
return parser.parse_args()
def canonicalize_cls(raw_cls):
if isinstance(raw_cls, str):
return raw_cls
if raw_cls is None:
return "<missing>"
try:
return json.dumps(raw_cls, ensure_ascii=False, sort_keys=True)
except TypeError:
return str(raw_cls)
def build_mapping_template(categories):
return {category: None for category in categories}
def main():
args = parse_args()
glob_pattern = args.glob_pattern or os.path.join(
args.json_root, "*", "annotations_20260320", "*.json"
)
json_files = sorted(glob.glob(glob_pattern))
if args.limit > 0:
json_files = json_files[: args.limit]
if not json_files:
print(f"未找到标注文件，pattern: {glob_pattern}")
return 1
os.makedirs(args.output_dir, exist_ok=True)
category_counter = Counter()
type_counter = Counter()
file_object_counter = Counter()
category_samples = {}
parse_errors = []
total_objects = 0
total_valid_camera_mea = 0
for json_path in json_files:
if '019b4bbf-a702-7c8d-abee-ec1ec17e66e9' in json_path or '019b4c4f-83c5-7aea-b364-cdaf93be8759' in json_path:
continue
try:
with open(json_path, "r", encoding="utf-8") as f:
data = json.load(f)
except Exception as exc:
parse_errors.append({"file": json_path, "error": str(exc)})
continue
objects = data.get("asso_list", [])
file_object_counter[json_path] = len(objects)
for index, obj in enumerate(objects):
total_objects += 1
camera_mea = obj.get("camera_mea")
if not isinstance(camera_mea, dict):
category = "<missing_camera_mea>"
raw_cls = None
else:
total_valid_camera_mea += 1
raw_cls = camera_mea.get("cls")
category = canonicalize_cls(raw_cls)
category_counter[category] += 1
type_counter[type(raw_cls).__name__] += 1
if category not in category_samples:
category_samples[category] = []
if len(category_samples[category]) < args.sample_per_category:
category_samples[category].append(
{
"file": json_path,
"index_in_asso_list": index,
"raw_cls": raw_cls,
"camera_mea_keys": sorted(camera_mea.keys()) if isinstance(camera_mea, dict) else [],
}
)
sorted_categories = [category for category, _ in category_counter.most_common()]
mapping_template = build_mapping_template(sorted_categories)
stats_report = {
"glob_pattern": glob_pattern,
"total_json_files": len(json_files),
"total_objects": total_objects,
"total_valid_camera_mea": total_valid_camera_mea,
"category_value_types": dict(type_counter.most_common()),
"category_counts": [
{"category": category, "count": count}
for category, count in category_counter.most_common()
],
"category_samples": category_samples,
"parse_errors": parse_errors,
}
report_path = os.path.join(args.output_dir, "category_stats.json")
mapping_path = os.path.join(args.output_dir, "category_mapping_template.json")
with open(report_path, "w", encoding="utf-8") as f:
json.dump(stats_report, f, ensure_ascii=False, indent=2)
with open(mapping_path, "w", encoding="utf-8") as f:
json.dump(mapping_template, f, ensure_ascii=False, indent=2)
print(f"统计完成，JSON 文件数: {len(json_files)}")
print(f"目标总数: {total_objects}")
print(f"有效 camera_mea 数: {total_valid_camera_mea}")
print("类别统计结果:")
for category, count in category_counter.most_common():
print(f" {category}: {count}")
print("cls 字段类型统计:")
for value_type, count in type_counter.most_common():
print(f" {value_type}: {count}")
if parse_errors:
print(f"解析失败文件数: {len(parse_errors)}")
print(f"详细统计已写入: {report_path}")
print(f"映射模板已写入: {mapping_path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

1458
tools/convert_gt_to_label/visualize_labels.py Executable file
View File

File diff suppressed because it is too large Load Diff