Several block structured languages (Scala, Go, Ruby, Julia, Quorum, ...) use semicolons as statement terminators, but allow newlines instead of semicolons under certain circumstances.
My question is: how can I represent Scala-like optional semicolons in a context free grammar? The specific issue is that some kinds of nesting delimiters "enable newlines" while others "disable newlines," so you need to pay attention to what kind of delimiters you are most immediately inside of.
I'm specifically asking about Scala's heuristic, because Julia and Quorum don't have specs. Ruby has a spec, but handles the problem by scattering "
[no line-terminator here]" throughout the formal grammar and I've been unable to find a general rule. Go has a well described heuristic but it's lexical only, which makes it obvious how to specify and implement, but its usability is somewhat disappointing. (Go inserts a semicolon even if you haven't closed the most recent
Scala goes well beyond the Go lexical rule with a nesting rule (cf reference, Section 1.2). In addition to lexically ensuring that both the token before and after the newline are consistent with the insertion of a statement terminator, newlines-as-statement-terminators are disabled between matching
) parenthesis and
] brackets, but then re-enabled between matching
I can figure out how to implement a simple pre-processor as a push-down automaton. The automaton starts with
enabled on the stack. As you process the token stream, when you see a
disabled onto the stack, when you see
enabled onto the stack, and when you see
} pop the top of the stack. Then when you see a newline that otherwise satisfies the lexical rules for the insertion of a statement terminator, you insert the terminator if and only if
enabled is on the top of the stack.
So Scala's newline-to-statement-terminator rule is "context free" in some sense, but I haven't been able to figure out how to incorporate this push-down automaton in with the rest of the language grammar.