Differentiable MPC-integrated MAPPO for multi-ship cooperative collision avoidance

Multi-ship collision avoidance requires simultaneous consideration of safety, COLREGs compliance, navigation efficiency, and coordinated decision-making under strongly coupled encounter situations. Reinforcement learning (RL) offers adaptive decision-making capabilities but has difficulty explicitly enforcing safety constraints, whereas model predictive control (MPC) provides systematic constraint handling but is often limited by predefined cost functions in complex and time-varying encounters. To combine their complementary strengths, this study proposes an improved multi-agent proximal policy optimization (MAPPO) framework integrated with differentiable model predictive control (D-MPC). A unified learning-constraint-optimization architecture is developed by embedding the D-MPC solver as a differentiable layer within the Actor network, allowing task-reward gradients to propagate through the optimization process and enabling end-to-end joint optimization of the policy and MPC cost parameters. A dynamic cost network (CostNet), coupled with gated recurrent units (GRUs), extracts temporal encounter features and adaptively adjusts the relative weights associated with collision avoidance and trajectory tracking. In addition, a dynamic reference-waypoint mechanism is introduced to explicitly incorporate guidance from the International Regulations for Preventing Collisions at Sea (COLREGs) into the optimization process. Experimental results show that, compared with conventional MAPPO and hierarchical RL-MPC baselines, the proposed method improves decision success rates by 80% and 30%, respectively, while reducing average path lengths by 6.9% and 11.3%. The results demonstrate that differentiable integration of multi-agent reinforcement learning and constrained predictive control can effectively improve the balance among safety, regulatory compliance, adaptability, and navigation efficiency in complex multi-ship encounters, while exhibiting favorable generalization under multi-objective conflict conditions.