Regular expression

Regular expressions are mathematical symbolism which describe the set of strings of specific language. It provides convenient and useful notation for representing tokens. Here are some rules that describe definition of the regular expressions over the input set denoted by ∑

 1) € is a regular expression that denotes the set containing empty string.                   

 2) If R1 and R2 are regular expressions then R=R1+R2 (same can also be represented as R=R1/R2)is also  regular expression which represents union operation.                                                                                                                  

 3) If R1 and R2 are regular expressions then R=R1.R2 is also a regular expression which represents concatenation operation.                                                                    

 4) If R1 is a regular expression then R=R1* is also a regular expression which represents kleen closure.

 A language denoted by a regular expression is said to be a regular set or regular language.

Read More

Error-recovery

Error-recovery actions are:

1)      Delete one character from the remaining input.

2)      Insert a missing character in to the remaining input.

3)      Replace a character by another character.

4)      Transpose two adjacent characters.

Read More

Lexical error

Lexical error include misspellings of identifiers, keywords, or operators 

Example:- the use of an identifier ellipse size instead of ellipse size and missing quotes around text intended as a string.

Read More

Tokens, patterns and lexemes

Token:  Token is a sequence of characters that can be treated as a single logical entity. Typical tokens are,                                                                                       

1) Identifiers 2) keywords 3) operators 4) special symbols 5)constants

Pattern: A set of strings in the input for which the same token is produced as output. This set of strings is described by a rule called a pattern associated with the token.

Lexeme: A lexeme is a sequence of characters in the source program that is matched by the pattern for a token.

Example:          

                              Description of token 

Token	lexeme	pattern
const	const	const
if	if	if
relation	<,<=,= ,< >,>=,>	< or <= or = or < > or >= or  letter followed by  letters & digit
i	pi	any numeric constant
nun	3.14	any character b/w “and “except"
literal	"core"	pattern

                                                                  

 A patter is a rule describing the set of lexemes that can represent a particular token in source program.

Read More

Issues in Lexical Analyizer

There are several reasons for separating the analysis phase of compiling in to lexical and parsing. 
1.Simpler design                                                                                                              

      2. Compiler efficiently is improved by specialized buffering techniques for reading input characters and processing tokens can significantly speed up the performance of a compiler.                                                                                                            

3. Compiler portability is enhanced

Read More

The Role of the lexical analyzer

                 The main task is to read the input characters and produce as output sequence of tokens that the parser uses for syntax analysis.

                         Fig 2.1 role of the lexical analyzer diagram 

 Up on receiving a “get next token” command from the parser, the lexical analyzer reads input characters until it can identify the next token.                                       

 Its secondary tasks are,                                                                                             

• One task is stripping out from the source program comments and white space is in the form of blank, tab, new line characters.                                                            
•  Another task is correlating error messages from the compiler with the source program.

Sometimes lexical analyzer is divided in to cascade of two phases.

1   1)  Scanning                   2) lexical analysis.

The scanner is responsible for doing simple tasks, while the lexical analyzer proper does the more complex operations.

Read More