0
$\begingroup$

I need to create a correct evaluation of the method I'm proposing. The problem is that in the field I'm working on there is no benchmark that I can use for this purpose. On the other hand, my method has to run, as much as possible, on top of "real documents", but these documents are mostly private and created by industry.

The documents are based on the AutomationML language, which is XML-based, and I do have some "real" ones. Until now, I have created the documents manually, and have also automatically generated them based on a given seed document and the AutomationML XML-Schema to ensure that the generated documents are syntactically correct. Also, they are generated following a uniform distribution.

I would like to extend the automatic generation but for this, I don't know how to automatically create the gold standard. I would appreciate any hints regarding this.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I'm not sure we have enough information here to answer your question. $\endgroup$
    – Yuval Filmus
    Apr 20, 2017 at 7:52
  • $\begingroup$ Can you point me to some strategies to Automatically generate testbeds of XML-based files along with their Gold Standards? Or I´m wrong here and Gold Standard should always be created by hand? $\endgroup$
    – Caleb
    Apr 20, 2017 at 7:58
  • $\begingroup$ Unfortunately I'm not an expert on this, and I'm not sure this question is on-topic here. $\endgroup$
    – Yuval Filmus
    Apr 20, 2017 at 8:01
  • $\begingroup$ But I think this is strongly computer science related, e.g., ontology matching, ontology alignment, etc, are very related to what I need to do. $\endgroup$
    – Caleb
    Apr 20, 2017 at 9:30

1 Answer 1

Reset to default
1
$\begingroup$

You can't. You can't squeeze blood out of a stone. If you want to evaluate how well your method will work on real documents, you need real documents. With synthetically/artificially generated documents, you'll never know if they have the same characteristics as real documents.

You don't necessarily need a standard benchmark that others have used -- but you do need to assemble a representative set of real documents to evaluate on. Basically, if there is no standard benchmark in your field, you'll have to assemble one yourself.

Yes, this is difficult to do. This is one of the less-well-known and less-glamorous aspects of working on machine learning: in practical projects, often we spend 80% of our time (or more!) just assembling data sets, and only 20% on the actual learning algorithms themselves.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Is there a way to ensure the validity of datasets? I have created manually and automatically the datasets but I'm having overfitting when applying my method. Could you suggest me something in this regards, some paper to read or so? $\endgroup$
    – Caleb
    May 10, 2017 at 8:43
  • $\begingroup$ " If you want to evaluate how well your method will work on real documents, you need real documents. With synthetically/artificially generated documents, you'll never know if they have the same characteristics as real documents." -- While true in the strict sense, all hope is not always lost. For instance, if the performance of your (graph) algorithm depends mostly on graph size and diameter, you can determine (or postulate) which parameters typically occur in your application, and then synthesize inputs with these parameters. Not perfect but better than nothing. $\endgroup$
    – Raphael ♦
    Nov 5, 2017 at 18:07
  • $\begingroup$ Of course, if you have no idea about the real inputs, you can explore the range of parameters to make general statements about your algorithm. "If scenario X, then use algorithm A. If Y, then use B." $\endgroup$
    – Raphael ♦
    Nov 5, 2017 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.