The Role of the lexical analyzer

                 The main task is to read the input characters and produce as output sequence of tokens that the parser uses for syntax analysis.

                         Fig 2.1 role of the lexical analyzer diagram 

 Up on receiving a “get next token” command from the parser, the lexical analyzer reads input characters until it can identify the next token.                                       

 Its secondary tasks are,                                                                                             

• One task is stripping out from the source program comments and white space is in the form of blank, tab, new line characters.                                                            
•  Another task is correlating error messages from the compiler with the source program.

Sometimes lexical analyzer is divided in to cascade of two phases.

1   1)  Scanning                   2) lexical analysis.

The scanner is responsible for doing simple tasks, while the lexical analyzer proper does the more complex operations.