Abstract: Currently, there is still no unified evaluation standard system for intelligent collision avoidance algorithms for vessels, either domestically or internationally. The development of an intelligent collision avoidance algorithm evaluation method that combines subjective and objective criteria is therefore of great significance for the realization of intelligent collision avoidance for vessels. This paper proposes a comprehensive evaluation method for the collision avoidance capability of two vessels in open water. Firstly, an algorithm evaluation system based on three core indexes of safety, compliance and economy is constructed using the reasonable quantification of the "International Regulations for Collision Avoidance at Sea" and experts' experience. Secondly, the weights of the indexes are determined using the hierarchical analysis method and the expert estimation method based on the results of the questionnaires collected, and an appropriate subordination function is established by choosing a suitable evaluation model and developing a fuzzy evaluation model for the intelligent collision avoidance algorithm. Secondly, an evaluation model and a fuzzy comprehensive evaluation system for the intelligent collision avoidance algorithm are developed. Finally, the constructed evaluation system is used to evaluate typical scenarios. The evaluation system is then used to evaluate typical scenarios. The test results demonstrate that the evaluation is fair and impartial, objectively reflecting the algorithm's strengths and weaknesses, and providing direction for improvement based on the scoring results.