Context-free grammar (CFG)

The grammars of programming languages these days are invariably specified using CFG (also called Backus-Naur Form (BNF)).
A parser is the part of the compiler that receives the stream of tokens from the lexical analyser (usually on a need-based basis) and returns a parse tree for valid input, or error message for invalid input.
Good syntax error handling is a very important and highly desirable property in a production compiler. A shining feature of clang is its specific and helpful error messages.
Possible strategies for error recovery are
- panic-mode
- phrase-level
- error productions
- global correction
Terms for CFG:
- terminals
- nonterminals
- start symbol
- productions
- left side or head of production
- right side or body of production
- alternatives

Notational Conventions

Terminals
- lowercase letters: \(a\), \(b\), \(c\)
- operator symbols: \(+\), \(-\), \(*\), \(/\), …
- punctuations: ( ) , ; :
- digits: 0, 1, …, 9
- boldface strings: id, if
Nonterminals
- \(A\), \(B\), \(C\)
- \(S\) is the start symbol
- lowercase italic names: expr, stmt
Uppercase letters late in the alphabet like \(X\), \(Y\), \(Z\), represent grammar symbols, i.e., either terminals or nonterminals.
Lowercase letters late in the alphabet like \(u\), \(v\), \(x\), \(y\), \(z\), represent strings of terminals.
Lowercase Greek letters like \(\alpha\), \(\beta\), \(\gamma\), represent strings of grammar symbols.

Derivations

Terms and Notations:
- \(\Rightarrow\) (“derive in one step”)
- \(\Rightarrow^*\) (“derive in zero or more steps”)
- \(\Rightarrow^+\) (“derive in one or more steps”)
- derivation
- sentential form
- sentence
- language \(L(G)\) generated by a grammar \(G\)
- \(\Rightarrow\atop lm\) (“leftmost derive in one step”)
- \(\Rightarrow\atop rm\) (“rightmost derive in one step”)
- \({\Rightarrow\atop lm}^*\) (“leftmost derive in zero or more steps”)
- \({\Rightarrow\atop rm}^*\) (“rightmost derive in zero or more steps”)
- leftmost sentential form
- rightmost sentential form

Ambiguity

A sentence is ambiguous if it’s the yield of two distinct parse trees; or equivalently, if it has two distinct leftmost derivations; or equivalently, if it has two distinct rightmost derivations.
A grammar is ambiguous if it produces some ambiguous sentence.

Verifying that a CFG generates the desired language

It’s important that the proposed grammar gives the desired language.
The only way to be certain that this is the case is to prove rigorously. The proof is invariably by induction: in one direction on the number of steps in the derivation of the sentence, and in the other direction on the length of the string of terminals. (See example in Ch. 4.2.6, p.204–205 of text.)

Writing a grammar

The syntax of a real programming language usually contains some context-sensitive features, e.g.,
- a variable needs to be declared before use
- the number and types of the actual arguments in a function call must match those of the parameters in the function definition
Thus, using CFG to specify the syntax will allow some invalid input to be accepted as valid. The compiler must check for these invalid sentences in the semantic analysis phase.
When writing the CFG, one has to check whether it’s ambiguous. If it is, there are two ways out.
1. Rewrite the grammar to make it unambiguous (but equivalent). (See Ch. 4.3.2, p.210–212.)
2. Use the ambiguous grammar but provide the parser with extra rules so it will return only the desirable parse tree.

Eliminating immediate left-recursion

Eliminating left-recursion is a necessary preprocessing step to make a grammar suitable for top-down parsing.
We first tackle the problem of eliminating immediate left-recursion. Given a nonterminal \(A\) with \(A\)-productions \[ A \to A\alpha_1 \mid A\alpha_2 \mid \cdots \mid A\alpha_m \mid \beta_1 \mid \beta_2 \mid \cdots \mid \beta_n \] where no \(\beta_i\) begins with an \(A\), we replace the \(A\)-productions with \[ A \to \beta_1A' \mid \beta_2A' \mid \cdots \mid \beta_nA' \] \[ A' \to \alpha_1A' \mid \alpha_2A' \mid \cdots \mid \alpha_mA' \mid \epsilon \]
Why?

Eliminating left-recursion in general

Assume the grammar has no \(\epsilon\)-production, i.e., production of the form \(A\to\epsilon\), and no cycle, i.e., production of the form \(A\Rightarrow^+ A\).

The following algorithm will eliminate immediate & non-immediate left-recursive productions.

arrange the nonterminals in some order A1, A2, ..., An  
for i ← 1 to n do {  
    for j ← 1 to i-1 do {  
        replace each production of the form Ai → Aj γ by  
          Ai → δ1 γ | δ2 γ | ... | δk γ, where  
          Aj → δ1 | δ2 | ... | δk are all current Aj-productions  
    }  
    eliminate immediate left-recursion among the Ai-productions  
}

Example of left-recursion elimination

Given the grammar
```
S → A a | b  
A → A c | S d | ε
```
we may order the nonterminals as S, A and apply the algorithm above. When i = 1, nothing happens. When i = 2, we substitute for S in A → S d to obtain the following A-productions
```
A → A c | A a d | b d | ε
```
Eliminating the immediate left-recursion for the A-productions gives the grammar
```
S → A a | b
A → b d A' | A'
A' → c A' | a d A' | ε
```
Even though the given grammar has an \(\epsilon\)-production, it turns out to be harmless in this example.
Exercise. Redo this example with the ordering A, S.

Left factoring

Left factoring is another necessary preprocessing step to make a grammar suitable for predictive or top-down parsing.
Suppose we have these \(A\)-productions \[ A \to \alpha \beta_1 \mid \alpha \beta_2 \mid \dots \mid \alpha \beta_n \mid \gamma \] where \(\alpha\) is the longest, nonempty, common prefix of the first \(n\) alternatives, and \(\gamma\) represents the remaining alternatives that do not begin with \(\alpha\). We can replace these \(A\)-productions with the productions \[ A \to \alpha A' \mid \gamma \] \[ A' \to \beta_1 \mid \beta_2 \mid \cdots \mid \beta_n \]
To left-factor the whole grammar, we repeat the above step to all nonterminals having alternatives with nonempty, common prefix until no nonterminal has two alternatives with a nonempty common prefix.

Example of left factoring

The grammar

S → i E t S | i E t S e S | a  
E → b

can be left-factered to

S → i E t S S' | a  
S' → e S | ε  
E → b

Context-free Grammar

San Skulrattanakulchai

February 21, 2019

Context-free grammar (CFG)

Notational Conventions

Derivations

Ambiguity

Verifying that a CFG generates the desired language

Writing a grammar

Eliminating immediate left-recursion

Eliminating left-recursion in general

Example of left-recursion elimination

Left factoring

Example of left factoring