1
$\begingroup$

I would like to get some opinions about using a context free grammar to generate sample utterances for Amazon Alexa/Google Assistant skills.

When developing these skills one has to provide a large number of sample utterances, i.e. sentences with which a user could try to invoke a particular feature of the skill. So when the skill is about ordering chocolate I would have to provide sentences like...

i would like to buy chocolate
i would like to order chocolate
we would like to buy chocolate
we would like to order chocolate

...and so on. the number of these sentences tends to grow almost exponentially, yet even with hundreds of them they still often become the bottleneck of the entire skill because users tend to come up with new formulations that weren't anticipated.

There are some tools available that can generate these sentences from simple patterns, i.e. this

{i|we} would like to {buy|order} chocolate

would produce the four variations given above. The drawback of this approach is that usually not all permutations of the pattern are valid (in most cases it is only a small subset of them) and one would thus need another "grammar" on top of these patterns to tell which are. The patterns are also not standardized and thus tied to one particular tool.

So I was looking for a better way to generate these sample utterances, and came across an approach that generates sentences from a context free grammar. This looks very promising to me, and in a small test script I was able to generate a large number of sentences from a very small grammar, with limitations on the sentences structure that I would have been able to include in the patterns of the existing tools.

I am however not a exactly an expert in Theoretical Computer Science and would like to ask a few questions before exploring this further:

  • My understanding is that a CFG is basically a "pattern" in the form of a tree. It would thus work similar to the patterns described above, but with the additional benefit of having the rules about which permutations of variables are legal embedded in the pattern itself. Is that correct?
  • If the CFG really is just a more complex pattern that would also mean that this approach (generating sentences from a CFG) can not produce grammatical variations of the generated sentences, e.g. it would not be able to generate pluralized forms like i would like to see _my_ shopping cart -> we would like to see _our_ shopping cart. Is that right?
  • More generally: From the hierarchy of formal languages, is a CFG really the best solution to this problem, our should I experiment with other types of grammars?

As I've said, it is entirely possible that I am misunderstanding some fundamental things about CFGs here, I am really just looking for some external feedback on this idea before investing more time in it. Thanks!

$\endgroup$
1
$\begingroup$

I don't think that will be effective. I don't recommend this approach. I think there's a significant risk that the machine learning will just learn to handle those specific synthetic variants, but not get any better at handling real speech. It might even get worse, if you cause the ML to overfit to your synthetic variants.

The problem is that real speech has a lot of variety in how people express things. Your synthetic examples are not going to capture that variety. Training machine learning on synthetic examples that aren't representative of what it will see during deployment is not likely to be effective. I suggest you try to obtain more realistic data, even if it can't be easily programmatically generated.

$\endgroup$
  • $\begingroup$ (1/2) I understand your skepticism, but I don't think your objection really refutes this idea. The speech-to-text transcription and the utterances matching are, from what all that is known, two separate steps. For the first one I would agree that a CFG isn't going to be of much help, but that step isn't actually under any control by the developer. I'm only talking about the text-to-text matching in the second step, whose input has the varieties of natural speech already filtered out. $\endgroup$ – gmolau Aug 21 '18 at 18:21
  • $\begingroup$ (2/2) In that step it is extremely important that both the general word order and the specific terms resemble the actual user input as closely as possible, and im only looking for a way to not have to write the same general sentence with 10 different verbs and 10 different synonyms for the noun (i.e. a hundred times), but still be able to specify the sentences more precisely than with a simple cartesian product. $\endgroup$ – gmolau Aug 21 '18 at 18:21
  • $\begingroup$ @gmolau, I know. I'm talking about the utterance matching part. I suspect those systems use machine learning for the utterance matching, and I worry this procedure will cause the model to overfit to the synthetic utterances you generated without necessarily getting better on real-world utterances. My point is that synthetic examples generated in this way won't resemble the distribution of data seen in actual user input; and I doubt there's any simple way to generate synthetic data that will. I am just speculating, of course. You can always try it and see if it works or not. $\endgroup$ – D.W. Aug 21 '18 at 21:18

This site is temporarily in read only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .