Reference selection for rating assignment based on pair comparison

Background: a set of 300 images are prepared, randomly pair compared with 60 other images, assigned a rating (from 1-star to 5-star) based on the pair comparison score (+1 if wins a pair comparison and +0 if loses a pair comparison), the score range for the rating assignment is 0-11 for 1-star, 12-23 for 2-star, 24-36 for 3-star, 37-47 for 4-star, and 48-60 for 5-star

Objective: want to rate 1000 more images based on the 300 rated images, again through pair comparison, but with some of the 300 rated images instead of between themselves

Problem:

• what images should be selected from the 300 rated images for the pair comparison, and how many?
• how should the score function be defined for the pair comparison with the selected image?

Could somebody help this experiment design? Thank you very much for you attention and help!

• – D.W.
Commented May 15, 2021 at 5:56
• @D.W. thank you very much for your suggestions! Commented May 16, 2021 at 11:58