Lexical Analysis is one of the most useful steps in compiler construction. A compiler is much more difficult to code than an interpreter, and so its design has to be broken up into a number of smaller modules, one of them being Lexical Analysis. The function of the lexical analyzer is to read the source code, one character at a time and to translate it into a sequence of primitive data units called tokens.
Need for Lexical Analysis :-
The objective of splitting the analysis phase of the source code into two separate sub-phases - lexical analysis and syntactic analysis - is to simplify the overall design of the compiler. The reason for this is that it is easier to specify the structure of tokens rather than the syntactic structure of the source code. Other important tasks of a lexical analyzer includes:
- Keeping track of line numbers
- Producing output listing
- Ignoring comments
- Removing unnecessary white spaces
Role of a Lexical Analyzer :-
The lexical analyzer is generally implemented as a separate pass, that is, it places its output to an intermediate temporary file which serves as input to the Parser which is the next logical step tool in compiler designs.
There are two ways of representing tokens. One way is to represent each token by an integer code if the token is a simple construct like a special character (< ; *). But if the token is more complex such as being a constant or an identifier, we can make a symbol table. The integer code in the table gives the type of the token, whereas the pointer points to a location which holds the value of the token under consideration.
References :-
- Compilers : Construction and Design by Aho, Sethi and Ullman
- System Programming by John Donovan
Written By Rae (5 January 2005)
Rae is a member of CAU Knowledge-Bank Tutorial Writers
This article was originally published by CyberArmy.net in the CyberArmy Library.
|
|