1
$\begingroup$

The paper mentions a line about GPT-3's training dataset -

Motivated by the ability of GPT-3 (Brown et al., 2020) to generate plausible code completions despite its training data not containing any explicitly gathered code datasets

My question - I need clarification on the following:

  1. If it was not trained on any "explicitly gathered code datasets", did the actual dataset on which it was trained on, contained some amount of code or exactly no code?

  2. If not, then how did it develop an ability to generate any code without learning what code really is?

$\endgroup$
2

1 Answer 1

2
$\begingroup$

Presumably "explicitly gathered" means that they didn't go out of their way to collect datasets specifically focused on code.

Undoubtedly the datasets they did use contained some code in it. The GPT3 paper says it was trained on a filtered version of CommonCrawl. This includes data from many sources that likely contain some code. For instance, Wikipedia contains code. So do some arXiv articles, some blog posts, some Stack Exchange answers, and many other sources.

$\endgroup$
1
  • $\begingroup$ 99% of the code that I write that ends up on the internet doesn’t work. (Because I try to tell people how to do it themselves; I’m not a code writing service). That’s probably the same for many developers. Good luck for any brainlessly text manipulation machine to create working code from this. $\endgroup$
    – gnasher729
    Feb 21 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.