InterPet4D Dataset Bridges Gap in Multimodal Human-Dog Interaction Modeling

A research team at the Institute of Science Tokyo has released InterPet4D, a multimodal 4D dataset designed to capture the complex dynamics of human-dog interactions. Presented at the 19th European Conference on Computer Vision on September 9, 2026, the dataset provides a foundation for training AI systems to interpret and respond to the physical and verbal signals exchanged between species.

The dataset comprises 6.83 million synchronized frames derived from 161 sessions involving 23 human participants and 13 dogs across 11 distinct breeds. Researchers utilized a configuration of 12 third-person cameras alongside egocentric video feeds to ensure comprehensive spatial coverage of the interactions. This multi-view approach addresses the technical challenge of occlusion, where one participant often obstructs the other during close-range activities.

The collected data is categorized into four primary interaction types: petting, commanding, calling, and free-form activities such as fetch or tug-of-war. Each session includes aligned audio and text captions, providing the necessary context for training models to associate specific human gestures or verbal cues with corresponding canine postural changes. The 4D aspect of the data allows for the analysis of movement trajectories over time rather than static snapshots.

Building upon this dataset, the team developed InterPetMoGen, an AI framework capable of generating plausible dog motion from human input. The architecture employs an autoregressive transformer that converts complex physical sequences into compact motion tokens for processing. A PetVAE autoencoder module handles the representation of canine movement, while a modality-aware attention mechanism integrates diverse input streams to inform the final output.

Performance benchmarks indicate significant improvements over existing baselines, such as the standard Seq2Seq-Transformer. InterPetMoGen achieved a kinetic Fréchet Inception Distance of 11.21, representing a 47.2% reduction in error compared to the 21.22 score of the baseline model. These metrics suggest that the framework produces motion sequences that more closely resemble real-world canine behavior.

Quantitative evaluations also demonstrated superior alignment between human cues and generated responses. Hand-motion alignment scores improved from 0.41 to 0.63, while body-motion alignment rose from 0.38 to 0.59. The model increased motion diversity from 5.01 to 5.93, indicating a broader range of generated responses to similar human stimuli, which is critical for creating realistic virtual agents.

The research team, led by Specially Appointed Assistant Professor Yichen Peng and Professor Hideki Koike, emphasizes that the dataset addresses a notable imbalance in computer vision research. Current models have historically prioritized human-human or human-object interactions, leaving cross-species communication largely underexplored. By standardizing the capture of these interactions, the researchers provide a mechanism for advancing behavioral analysis and the development of socially aware robotics.

Subjective assessment through a user study involving 12 participants confirmed these technical gains. The model received high ratings for naturality, responsiveness, and overall quality on a seven-point scale, significantly outperforming previous approaches. These results validate the importance of incorporating broader interaction contexts when training generative models for non-human subjects in machine learning environments.

The current iteration serves as a benchmark for future work in virtual agent animation and pet-centric AI applications. By capturing the multimodal nature of human-dog communication, the framework supports more systematic research into how AI can interpret non-verbal cues in real-time. This standardized foundation is essential for moving beyond simple motion synthesis toward more nuanced, context-aware robotic responses that can function effectively in domestic settings.

Future iterations of the research aim to overcome current limitations, such as the inability to account for physical contact forces and the restriction to fixed-length 10-second clips. Expanding the framework to include longer sequences and diverse species remains a primary objective for the team. These advancements will likely refine the precision of motion generation in increasingly complex, real-world environments.