5.9 KiB
DWPose
Whole-body pose estimation localizes the human body, hand, face, and foot keypoints in an image. This task is challenging due to multi-scale body parts, fine-grained localization for low-resolution regions, and data scarcity. Meanwhile, applying a highly efficient and accurate pose estimator to widely human-centric understanding and generation tasks is urgent. In this work, we present a two-stage pose Distillation for Whole-body Pose estimators, named DWPose, to improve their effectiveness and efficiency. The first-stage distillation designs a weight-decay strategy while utilizing a teacher's intermediate feature and final logits with both visible and invisible keypoints to supervise the student from scratch. The second stage distills the student model itself to further improve performance. Different from the previous self-knowledge distillation, this stage finetunes the student's head with only 20% training time as a plug-and-play training strategy. For data limitations, we explore the UBody dataset that contains diverse facial expressions and hand gestures for real-life applications. Comprehensive experiments show the superiority of our proposed simple yet effective methods. We achieve new state-of-the-art performance on COCO-WholeBody, significantly boosting the whole-body AP of RTMPose-l from 64.8% to 66.5%, even surpassing RTMPose-x teacher with 65.3% AP. We release a series of models with different sizes, from tiny to large, for satisfying various downstream tasks.
Results and Models
COCO-WholeBody Dataset
Results on COCO-WholeBody v1.0 val with detector having human AP of 56.4 on COCO val2017 dataset
- DWPose Models are supported by DWPose
- Models are trained and distilled on the following datasets:
| Config | S1 Dis_config | S2 Dis_config | Input Size | Whole AP | Whole AR | FLOPS (G) |
ORT-Latency (ms) (i7-11700) |
TRT-FP16-Latency (ms) (GTX 1660Ti) |
Download |
|---|---|---|---|---|---|---|---|---|---|
| DWPose-t | DW l-t | DW t-t | 256x192 | 48.5 | 58.4 | 0.5 | - | - | Model |
| DWPose-s | DW l-s | DW s-s | 256x192 | 53.8 | 63.2 | 0.9 | - | - | Model |
| DWPose-m | DW l-m | DW m-m | 256x192 | 60.6 | 69.5 | 2.22 | 13.50 | 4.00 | Model |
| DWPose-l | DW x-l | DW l-l | 256x192 | 63.1 | 71.7 | 4.52 | 23.41 | 5.67 | Model |
| DWPose-l | DW x-l | DW l-l | 384x288 | 66.5 | 74.3 | 10.07 | 44.58 | 7.68 | Model |
Train a model
Train DWPose with the first stage distillation
bash tools/dist_train.sh configs/wholebody_2d_keypoint/dwpose/ubody/s1_dis/rtmpose_x_dis_l_coco-ubody-384x288.py 8
Tansfer the S1 distillation models into regular models
# first stage distillation
python pth_transfer.py $dis_ckpt $new_pose_ckpt
⭐Before S2 distillation, you should add your model path into 'teacher_pretrained' of your S2 dis_config.
Train DWPose with the second stage distillation
bash tools/dist_train.sh configs/wholebody_2d_keypoint/dwpose/ubody/s2_dis/dwpose_l-ll_coco-ubody-384x288.py 8
Tansfer the S2 distillation models into regular models
# second stage distillation
python pth_transfer.py $dis_ckpt $new_pose_ckpt --two_dis
Citation
@article{yang2023effective,
title={Effective Whole-body Pose Estimation with Two-stages Distillation},
author={Yang, Zhendong and Zeng, Ailing and Yuan, Chun and Li, Yu},
journal={arXiv preprint arXiv:2307.15880},
year={2023}
}