Real World Perception: Sensor Fusion for BJJ Pose Estimation
Executive Summary
I. Real World Perception: Sensor Fusion for BJJ Pose Estimation
Video, tactile contacts, joint encoders, and inertial measurement units (IMUs) are fused to estimate body position, orientation, and motion. The perception state also considers center of mass, inertia, momentum, detected resistance, and grip position and strength, distinguishing direct measurements from model-based estimates and retaining uncertainty.
A. BJJ Pose Recognition
BJJ pose recognition begins by estimating where both bodies are and how their articulated regions are arranged. The proposed architecture combines camera observations with the robot’s joint encoders, inertial measurements, and available contact and force measurements. The robot has direct measurements of its own joint configuration; the participant’s configuration must be inferred from observations and interaction.
A useful representation includes the position and orientation of the torso, pelvis, head, arms, and legs; estimated joint configurations; motion over time; and each body’s relationship to the mat. Observations are aligned to a common coordinate frame and time reference so that a detected wrist, a measured grip force, and a change in robot joint position can be interpreted together.
The system can use these estimates to identify candidate BJJ configurations such as closed guard, half guard, side control, mount, or back control, including which participant occupies each role. Position recognition should retain confidence levels and competing hypotheses. A familiar silhouette alone does not establish control: two visually similar configurations may have different grips, trapped limbs, or routes of escape.
Occlusion is a central challenge. A hand may disappear behind a torso, or a leg may be hidden beneath both bodies. Temporal tracking, contact observations, and kinematic constraints can help maintain an estimate of the hidden region. The system should distinguish observed state from inferred state and retain uncertainty when the available evidence cannot resolve the position.
The output is therefore more than a position label. It is a continuously updated estimate of both bodies, their motion, candidate BJJ positions, and the uncertainty associated with those estimates. That estimate provides the geometric foundation for inter-body intelligence.
VIDEO AND TACTILE CONTACTS: A FUSED POSE
Video estimates visible body geometry. Tactile sensors detect contact at the robot’s hands, thighs, and legs; available measurements may include pressure and shear. Joint encoders locate the robot’s sensing surfaces in space. Time-aligned fusion constrains configurations consistent with both visible geometry and sensed contacts.
Contact alone does not identify the opponent body region touching the sensor. Visual evidence, geometry, and recent movement help associate the contact with an arm, shoulder, torso, or leg. Hidden regions remain inferred, with confidence and competing interpretations retained.
B. Robot Sensors and Physical State
The proposed BJJ humanoid uses distributed sensing to maintain an estimate of its own physical state and its contact with the training partner and mat. Sensor locations and coverage depend on the selected platform and its grappling adaptations.
Video cameras: head- or body-mounted cameras observe the partner, visible robot limbs, and mat. Visual tracking contributes body geometry, relative position, and motion; calibration relates camera observations to the robot’s joint and contact measurements.
Body and finger sensors: tactile sensing on fingertips, palms, forearms, torso, thighs, and shins can report contact location and pressure, with shear or slip information where supported. Finger-position and force measurements help estimate grip configuration, applied gripping force, and whether contact is slipping. Clothing and padding affect how these measurements are interpreted.
Joint and inertial sensing: joint encoders report articulated configuration, while available joint-torque or force sensors contribute loading and resistance information. IMUs on the torso or selected limbs measure acceleration and angular rate. Fusing these observations supports orientation, velocity, and position estimates; absolute position requires additional references to limit drift.
Balance and support sensing: foot-pressure measurements can estimate pressure distribution and center of pressure during foot contact. During ground grappling, support may also occur through knees, hands, forearms, torso, or the partner, so the estimator considers all detected supporting contacts rather than assuming a standing stance.
Physical-state output: synchronized sensor observations are combined with the robot’s geometry, mass distribution, and inertia model to estimate body and joint position, motion, center of mass, momentum, support, applied loads, grip condition, and detected resistance. Each estimate retains its time reference and confidence. Partner properties remain estimates with their own uncertainty.
This physical-state representation feeds section II’s inter-body artificial intelligence and BJJ LLM, helping it evaluate candidate actions and generate BJJ action tokens. Section III trains and adapts that LLM to the instructor’s persona and methodology.
C. Physics-Aware State Estimation
Position and inertial sensing: camera geometry and robot joint encoders estimate body and limb positions. IMUs provide acceleration and angular-rate measurements that contribute to orientation and motion estimates. Measurements are aligned in time and coordinate frame; an IMU alone does not provide drift-free absolute position.
Center of mass, inertia, and momentum: the system combines estimated pose and velocity with body-segment mass and inertia models to estimate weight distribution and linear and angular momentum. Robot properties can come from its model; participant properties require estimates with uncertainty. These quantities help characterize balance, support, and how motion may continue as contact changes.
Resistance and grip: tactile pressure, shear, force or torque measurements, and observed movement provide evidence of resistance and grip position. Grip strength is estimated from available force and slip evidence rather than inferred from visual contact alone. Resistance may arise from active opposition, friction, inertia, or a mechanical constraint; the estimator retains ambiguity when those causes cannot be distinguished.
The output passed to section II is a fused pose and physical-state estimate: body positions and motion, contact and grip observations, support and center-of-mass estimates, inertia and momentum estimates, detected resistance, and confidence. Inter-body artificial intelligence interprets these inputs to evaluate actions.
Related architecture details
Proposed Architecture
Technical Specifications
Partner with
TrustRobotics
Integrate the industry's premier
AI-to-actuator safety layer.
Accelerate Open Development
Validate & Test Stack
Secure Commercial Licensing