Abstract

Prolonged mid-air interaction in virtual reality (VR) can lead to physical discomfort and arm fatigue, commonly referred to as the gorilla arm effect. This phenomenon negatively affects user performance and experience, highlighting the need for ergonomically informed interaction design in immersive systems.
Incorporating ergonomic considerations into VR user interface (UI) design typically requires extensive human-in-the-loop data collection to assess physical effort and fatigue. Such evaluations are costly, time-consuming, and difficult to scale, limiting their usefulness during early-stage design exploration. As a result, there is a need for alternative methods that can support ergonomic evaluation without relying exclusively on repeated human studies.
Biomechanical user models offer a promising direction, as they represent complex, muscle-actuated human motion and can estimate physical effort and fatigue during interaction. While these models have been used to simulate human movement and behavior in human–computer interaction (HCI) research, their potential role as surrogate evaluators for ergonomic VR UI design remains largely unexplored.
This work presents a framework that leverages biomechanical user models as surrogate users to evaluate and optimize VR interfaces for mid-air interaction. A biomechanical motion agent is trained using reinforcement learning (RL) to perform sequential button-press tasks in VR, generating realistic movement strategies and estimating muscle-level fatigue using a validated three-compartment controller with recovery (3CC-r) fatigue model. The simulated fatigue estimates are then used as a feedback signal for an interface optimization agent that learns spatial UI layouts through RL, with the objective of minimizing cumulative user fatigue.
The approach is evaluated by comparing RL-optimized layouts against a manually designed centered baseline and a Bayesian optimized baseline. Results show that fatigue trends predicted by the biomechanical model align with those observed in a human user study, and that interfaces optimized using RL lead to lower perceived fatigue. The framework’s extensibility is further demonstrated through a use case involving longer sequential tasks with non-uniform interaction frequencies.