One of the first jobs performed by a compiler is to read in the source code and group individual characters into tokens., the lowest level of symbols used in defining a programming language.
A C++ compiler always collects characters into the longest possible tokens, using white spaces to demarcate them. White spaces must be used to seperate an identifier, reserved word, integer constant, floating point constant from a following identifier, reserved word, integer constant or floating point constant and arithmetic or boolean operators.
The tokens available in C++ are: identifiers, constants, punctuators, operators and variables.
Identifiers
Symbolic names used for various data items by a programmer in a program are known as identifiers. An identifier is a sequence of characters and digits and / or underscore. It represents a memory location. C++ sets no maximum length for an identifier through most compilers allow 31 characters.
Rules for naming an identifier:
(a) The first character of anidentifier must be a letter.
(b) It must not start with a digit.
(c) An identifier can start with an underscore. An underscore counts as a letter.
(d) The name of the identifier should be such that it is relevant to its use in the program.
(e) C++ is case-sensitive. Upper-case and lower-case letters are considered different from each other.
Example of Valid identifiers: ROLL, Total, Basic_sal, _sturec, ob1, marks.
Examples of invalid identifiers: Emp rec (blank not allowed), class,avg (comma not allowed)
Keyword
A keyword is a word carrying special meaning and purpose for a language. The following identifiers are reserved for use by C++ as keywords. Keywords cannot be used as an identifier by the user in his program. The set of keywords is givenin the following table:
Constants ( or Literals)
A data item which does not change its value during the execution of a program is known as a literal. They are often reffered to as constants. A constant can be defiend in any of the various basic data types. The const qualifier is used to declare a constant.
Integer Constant
An Integer constant or literal consists of a sequence of digits. An Integer constant can be assigned integer values only. An integer literal takes different forms. A decimal integer literal (of base 10) is a non-zero digit followed by zero or more decimal digits. A hexadecimal integer literal (of base 16) is 0x or 0X folloed by a sequence of one or more hexadecimal digits. An octal integer literal (of base 8) is the digit 0 followed by a sequence of zero or more octal digits.
Character Constant
A character constant consists of one or more characters enclosed in single quotes. For example, 'X' , '\t' and '\032' are character constants.
Floating point constant
In a decimal floating literal, the exponent part is optional. If presnt, it begins with an E or e followed by decimal digits representing a power of 10.
String Literal
A string literal is a sequence of zero or more characters surrounded by double quotes. When a string literal is assigned to an identifier as a constant, then it is known as a string constant.
Punctuators
The ASCII representatives of C++ programs uses the following characters as operators or for punctuation. These characters are also known as seperators.
[ ] ( ) { } , ; : * = #