I'm doing a research on NLG systems. I need to annotate my corpus (~6 million words) automatically. My algorithm works well and I want to calculate Cohen's Kappa. What I cannot understand is the second annotator; I don't have the possibility to have another annotation. Is there any solution for this problem? Can I have a set of data as my sample and manually doing the annotation to calculate Cohen's Kappa? Is there any measurement for corpus analysis?