Lexical Analyzer and Operator Grammar


Hey all, Welcome to Geeksforjobs

What is Lexical Analyzer?

The lexical analyzer is implemented as a sub-routine. Whenever the parser needs a new token, it is called Lexical Analyzer. Next, we need a literal analyzer for token recognition.

What is a token?

Tokens can be different types of identifiers, punctuation marks, logical operators, arithmetic operators and so on. This keyword can also be start, end, if etc.

How does it analyze tokens?

  • There is also a special buffer which is known as an input buffer.
  • A statement of the high-level language program and the pointer was placed, known as the token start (this points to the token as we have previously recognized a token).
  • Next there will be another look pointer and it will proceed by character and check if there is a delimiter (it indicates the end of a particular token).
  • Therefore it is also known as a token.
What does Lexical Analyzer do?
  • 1.Limim to Token
  • 2. Playing white whatsapp
  • 3.Removing Comments
  • 4. Correcting Errors

Note: -Sexual Analyzer is reading line by line. It is reading line by line because it needs to convert all the lemons into tokens.

How to count tokens?

int max (x, y)

int x, y;

/ * Find the maximum of x and y * /


Return (x> y x: y?);


The tokens are: –

We will count each character one by one.

1.int 2.max 3. ( 4.x 5.,
6.y 7.; 8.) 9.int 10.x
1 1., 12.y 13. { 14.return 15. (
16.x 17.> 18.y 19.? 20.x
21 .: 22.y 23.) 24.; 25.}
Token count

The total tokens are 25.

  • Number of tokens in the following C statement: –
    • Printf (“% d hello”; i; & i); is

The tokens are: –

We will count each character one by one.

But there is a twist, a string of tokens so it will be considered one. This means, “% d hello” – the string of the token will be taken as one.

1.printf 6.;
2. ( 7. More
3. “% D hello” 8.i
4.; 9.)
5. I 10.;
Token count

Total tokens are 10.

What is operator grammar?
  • A grammar that is commonly used to define mathematical functions.
  • There is no restriction in grammar or the operator cannot be in grammar: –
  • Any two adjacent non-terminals on the RHS of production.
  • Null variable
  • E -> E + E / E * E / ID [Operator Grammar]
    • This output will be an operator grammar because it does not violate the two rules.
  • E -> EAE / ID
  • A -> + / *
    • This will not be a production operator grammar because it is the first rule to have two adjacent non-terminals on the RHS side.
    • Therefore, we first need to convert it into operator grammar by placing the values ​​of ‘A’ in the output.
  • E -> E + E / E * E / ID [Operator Grammer]

Previous articleEducation..
Next articleSoftware Development Engineer Intern at Amazon

Leave a Comment

Your email address will not be published. Required fields are marked *