Use different types of prompts

[cloglove]

I would like to use different types of prompts to replace the keypoints, such as human part masks generated by SCHP. Is this approach feasible?

[VlSomers]

Hi @cloglove , yes this approach is feasible! You would just need to change how prompts are read and processed, let me know if you need more info

[cloglove]

Hi @VlSomers, thank you for the confirmation. I have successfully generated .pngformat human parsing masks for the Market-1501 dataset using SCHP. However, due to my limited expertise, I would greatly appreciate some further guidance on the subsequent reading and processing steps, as I currently only have the .pngformat masks. Thank you again for taking the time to answer my questions.

[VlSomers]

Hi, would you like to use the model as is with your masks? If you want to use other types of prompts, you will need to retrain the model. You will need a training dataset and mimic what has been done in other datasets (e.g. torchreid/data/datasets/image/occluded_dukemtmc.py) to load your human parsing masks. Are you ok with retraining?

[cloglove]

Thank you for your reply. I have processed the human body masks generated by SCHP and obtained double-channel maps and per-pixel labels to replace the original M (keypoint heatmaps) and human parsing labels, respectively. Is this processing correct? Can it be used for training?

[VlSomers]

yes, you then need to create your own dataset, e.g. have a look at torchreid/data/datasets/image/occluded_dukemtmc.py. You need to point the masks_path to you SCHP masks. You can have a look at torchreid/data/transforms.py to see how masks are transformed and handled. You can also have a look at torchreid/data/datasets/keypoints_to_masks.py. Finally, you mainly need to have a look at torchreid/data/datasets/dataset.py to understand how masks are return next to each image sample via the getitem function (used at training and testing time).

[cloglove]

Hi, I replaced KPR’s keypoint heatmap prompts with SCHP-based dual-channel pixel-level mask prompts, but mAP and Rank-1 dropped on Market1501.Do you think this replacement itself is unlikely to improve KPR significantly, or could the performance drop be caused by my implementation?I would really appreciate your guidance and any corrections or suggestions.

[VlSomers]

I tried SCHP in the past and it produced very bad results, and it will be even worse in occluded scenario. Moreover, if you produce SCHP masks for occluded scenarios with multiple person and it works, you still need to manually select the real target among the multiple persons. Why do you want to use SCHP instead of a pose estimation model?

[cloglove]

I was thinking about whether pixel-level masks could be used in person re-identification to obtain more detailed features. When I came across KPR, it seemed that your framework could support this idea, so I decided to explore it. SCHP is only a tool I used to generate fine-grained masks and labels for this experiment.

In a small ablation experiment, I found that replacing the real dual-channel masks with all-zero masks produced exactly the same evaluation results. This makes me wonder whether the prompt branch is actually contributing to the model.

Therefore, I would like to ask whether replacing keypoints with dual-channel masks is a feasible and potentially useful direction.