Multimodal Contrastive Learning for Zero-Shot Instruction-Following Robot with Synthetic Data

Robot trajectory prediction frequently necessitates extensive real-world demonstrations, which introduces challenges related to scalability, data acquisition costs, and the achievement of zero-shot generalization. This research introduces Zero-Shot Task Learning (ZSTL), a multimodal framework engineered to facilitate instruction-based trajectory generation without reliance on real-world demonstration data. ZSTL employs structurally aligned synthetic data and contrastive learning to achieve its objectives. The framework jointly encodes natural language instructions, depth observations, LiDAR-derived spatial representations, and corresponding action trajectories within a unified embedding space. This integration enables robust cross-modal alignment and conditional behavior synthesis. The proposed architecture preserves the inherent structure of modalities by representing depth inputs as spatial tokens and LiDAR observations as temporal

tokens. These, combined with a text token, form a 101-token multimodal context. A Transformer decoder then processes this context to predict full 50-step trajectories, complete with Gaussian uncertainty estimates. The system incorporates a pre-trained Bidirectional Encoder Representations from Transformers (BERT) language encoder, a ResNet-18 depth backbone, and a one-dimensional convolutional LiDAR sequence encoder, all feeding into a two-layer Transformer decoder comprising approximately 125 million parameters. Training was exclusively performed on a procedurally generated synthetic dataset of 5,000 samples over 50 epochs.

Message: Hi FE, I need help with “Multimodal Contrastive Learning for Zero-Shot Instruction-Following Robot with Synthetic Data” in “Computer Science & Engineering”