Multi-animal pose estimation, identification and tracking with DeepLabCut

Abstract Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar looking animals that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, an open-source pose estimation toolbox, and provide high-performance animal assembly and tracking—features required for multi-animal scenarios. Furthermore, we integrate the ability to predict an animal’s identity to assist tracking (in case of occlusions). We illustrate the power of this framework with four datasets varying in complexity,…

• HH
  Howard Hughes Medical Institute
• HU
  Harvard University
• FB
  Fondation Bertarelli
• CZ
  Chan Zuckerberg Initiative
• ÉP
  École Polytechnique Fédérale de Lausanne
• SN
  Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
• NI
  National Institutes of Health
• RI
  Rowland Institute at Harvard
• OO
  Office of Naval Research

  Awards: N000141410533, N00014-15-1-2234, N00014