# context-free grammar for comma-separated list with comments

I'm trying to write a context-free grammar for a comma-separated list of statements with comments. It is trivial if the comma appears at the end of a statement. It is trivial if the comment is ignored. But I don't know how to include a comma in the middle of a statement (before the comment).

This Python / Lark CFG:

start: def+

def: "{" statement ("," statement)* "}"

!statement: NAME "=" (
| "Foo"
| "Bar" "(" NAME ")"
| "Tomato"
| )

NAME: /[A-Z][a-z-]{0,15}/

COMMENT: "//" /.*/

%import common.WS
%ignore WS


parses this text:

{
Mike = Foo,
Fred = Bar(Chocolate),
Joe = Tomato
}


producing this AST:

start
def
statement
Mike
=
Foo
statement
Fred
=
Bar
(
Chocolate
)
statement
Joe
=
Tomato


I want to parse the content of comments into the AST, not just ignore it:

{
Mike = Foo,
Fred = Bar(Chocolate),      // Yummy!
Joe = Tomato
}


======= EDIT =========

The problem from which the question was distilled requires an optional description tied to each assignment. Arbitrary comments (such as on the lines with braces) aren't needed. So Yuval's answer (with the addition of a trailing comment) works like a charm. For completeness, the working grammar is:

start: def+
def: "{" statement ("," COMMENT? statement)* COMMENT? "}"

statement: NAME "=" (foo | bar | tomato)
foo:    "Foo"
bar:    "Bar" "(" NAME ")"
tomato: "Tomato"


and produces the AST:

start
def
statement
Mike
foo
// Stuff
statement
Fred
bar   Chocolate
// Yummy!
statement
Joe
tomato
// aaaargh

• This sounds like a coding issue more than a CFG issue – D. Ben Knoble May 20 '19 at 17:17
• Comments are usually handled differently – they are simply removed by the lexical analyzer. – Yuval Filmus May 20 '19 at 17:36
• Yes, as acknowledged in the question. I'm not asking what is usual, I'm asking whether the unusual is possible (within some threshold of additional complexity/usability/understandability). – Dave May 20 '19 at 17:42
• @D.BenKnoble: Using any CFG notation you like, parse the text at the bottom into an AST that includes the comment. The question has nothing to do with coding, even though a code example is shown for concreteness. – Dave May 20 '19 at 19:53

Define a non-terminal COMMENT which generates comments (in your case, // followed by arbitrary non-newline characters and then newline), as well as the empty string. In the definition of def, add COMMENT after ,.