Tokens, lexemes, and patterns(MCQs)

What is a “token” in the context of lexical analysis? a) A sequence of machine code instructions b) A basic unit of syntax such as keywords, operators, or identifiers c) A type of syntax error d) An intermediate code representation Answer: b) A basic unit of syntax such as keywords, operators, or identifiers What does a “lexeme” represent in lexical analysis? a) A sequence of characters that forms a token b) A machine code instruction c) A syntax error d) An intermediate code structure Answer: a) A sequence of characters that forms a token What is a “pattern” in lexical analysis? a) A set of syntax rules for parsing code b) A regular expression that defines token types c) A sequence of machine code instructions d) A type of runtime error Answer: b) A regular expression that defines token types Which component uses patterns to identify tokens in source code? a) Syntax analyzer b) Lexical analyzer c) Code generator d) Optimizer Answer: b) Lexical analyzer In lexical analysis, what is the relationship between a token and its lexeme? a) A token is the representation of a lexeme b) A lexeme is the representation of a token c) They are the same thing d) A token is a subset of a lexeme Answer: b) A lexeme is the representation of a token Which of the following is an example of a token? a) An identifier like variableName b) A sequence of characters in a string literal c) A regular expression pattern d) A syntax error message Answer: a) An identifier like variableName What is the primary purpose of using patterns in lexical analysis? a) To define how tokens are recognized and classified b) To generate machine code c) To create syntax trees d) To optimize code Answer: a) To define how tokens are recognized and classified Which of the following is NOT a type of token? a) Keyword b) Identifier c) Regular expression d) Operator Answer: c) Regular expression What is the role of regular expressions in defining patterns for tokens? a) They specify the format and structure of tokens b) They generate intermediate code c) They parse tokens into syntax trees d) They optimize the code Answer: a) They specify the format and structure of tokens How does a lexical analyzer use lexemes? a) To generate intermediate code b) To identify and classify tokens based on patterns c) To perform semantic analysis d) To optimize code performance Answer: b) To identify and classify tokens based on patterns What is an example of a lexeme? a) if in a source code b) An assembly language instruction c) A machine code instruction d) An error message Answer: a) if in a source code What is the function of a “token table” in lexical analysis? a) To store patterns for token recognition b) To manage recognized tokens and their attributes c) To generate syntax trees d) To perform semantic analysis Answer: b) To manage recognized tokens and their attributes Which of the following best describes a “pattern” used in lexical analysis? a) A sequence of machine code instructions b) A set of rules for recognizing token types c) A type of runtime error d) A format for generating intermediate code Answer: b) A set of rules for recognizing token types What does the lexical analyzer do with a lexeme? a) Converts it into machine code b) Recognizes it as a specific token type c) Creates an abstract syntax tree d) Performs code optimization Answer: b) Recognizes it as a specific token type How are tokens typically represented in the lexical analysis phase? a) As machine code b) As a sequence of characters c) As a combination of token type and value d) As syntax error messages Answer: c) As a combination of token type and value Which of the following best describes the role of patterns in defining tokens? a) They specify how tokens are recognized and categorized b) They generate syntax trees c) They perform semantic analysis d) They create machine code Answer: a) They specify how tokens are recognized and categorized What is a common use of regular expressions in lexical analysis? a) To define token patterns b) To generate intermediate code c) To create syntax trees d) To perform code optimization Answer: a) To define token patterns Which tool is often used to generate lexical analyzers from regular expressions? a) Yacc b) Bison c) Lex d) Gcc Answer: c) Lex In lexical analysis, what is meant by “token classification”? a) Identifying and categorizing tokens based on patterns b) Generating machine code c) Parsing tokens into syntax trees d) Performing semantic analysis Answer: a) Identifying and categorizing tokens based on patterns What is an example of a pattern that might be used to recognize a token? a) [0-9]+ for recognizing integers b) if for recognizing keywords c) * for recognizing operators d) = for recognizing assignment statements Answer: a) [0-9]+ for recognizing integers What is the role of a lexeme in the tokenization process? a) To serve as the actual sequence of characters that matches a pattern b) To generate machine code c) To parse tokens into syntax trees d) To perform code optimization Answer: a) To serve as the actual sequence of characters that matches a pattern What does the lexical analyzer do when it encounters a string literal? a) Converts it into a token representing the string b) Generates a syntax tree c) Creates intermediate code d) Ignores it Answer: a) Converts it into a token representing the string How does a lexical analyzer distinguish between different token types? a) By using patterns defined by regular expressions b) By performing syntax analysis c) By generating machine code d) By creating abstract syntax trees Answer: a) By using patterns defined by regular expressions What is an example of a token type that a lexical analyzer might identify? a) Keyword b) Variable name c) Operator d) All of the above Answer: d) All of the above What is the purpose of “token attributes” in lexical analysis? a) To provide additional information about tokens b) To define token patterns c) To generate machine code d) To create syntax trees Answer: a) To provide additional information about tokens Which of the following is not typically considered a pattern for token recognition? a) Regular expressions b) Finite state machines c) Context-free grammars d) Syntax trees Answer: d) Syntax trees What does the term “token stream” refer to in lexical analysis? a) A sequence of recognized tokens and their attributes b) A list of syntax errors c) A sequence of machine code instructions d) A set of intermediate representations Answer: a) A sequence of recognized tokens and their attributes How does a lexical analyzer handle invalid tokens? a) It generates an error message b) It skips them c) It attempts to correct them d) It ignores them Answer: a) It generates an error message Which tool is used to specify patterns for lexical analysis using regular expressions? a) Yacc b) Bison c) Lex d) Gcc Answer: c) Lex What is the main function of “tokenization” in lexical analysis? a) To convert source code into a sequence of tokens b) To generate machine code c) To create syntax trees d) To perform semantic analysis Answer: a) To convert source code into a sequence of tokens Which of the following components in lexical analysis helps in identifying patterns? a) Finite state machines b) Syntax analyzers c) Code generators d) Optimizers Answer: a) Finite state machines What does the term “lexical error” refer to? a) An error in recognizing valid tokens b) An error in syntax analysis c) An error in code optimization d) An error in code generation Answer: a) An error in recognizing valid tokens What is the purpose of a “token stream” in the compilation process? a) To provide the lexical analyzer’s output to the next phase of compilation b) To generate machine code c) To create syntax trees d) To perform semantic analysis Answer: a) To provide the lexical analyzer’s output to the next phase of compilation Which of the following best describes a “finite state machine” used in lexical analysis? a) A model that processes input sequences to recognize tokens b) A tool for generating intermediate code c) A method for optimizing code d) A parser for syntax analysis Answer: a) A model that processes input sequences to recognize tokens What is the role of “comment removal” in lexical analysis? a) To ignore non-essential parts of the code that are not tokens b) To handle multi-character tokens c) To generate machine code d) To parse tokens into syntax trees Answer: a) To ignore non-essential parts of the code that are not tokens Which of the following is a typical output of the lexical analyzer? a) A sequence of tokens b) Machine code instructions c) Syntax trees d) Intermediate code Answer: a) A sequence of tokens How does the lexical analyzer handle whitespace in the source code? a) It typically ignores whitespace unless it is part of a token b) It converts it into tokens c) It generates syntax errors for excessive whitespace d) It performs semantic analysis on whitespace Answer: a) It typically ignores whitespace unless it is part of a token What is an example of a pattern used to recognize an identifier token? a) [a-zA-Z_][a-zA-Z0-9_]* b) [0-9]+ c) “[^”]*” d) [+\-*/] Answer: a) [a-zA-Z_][a-zA-Z0-9_]* Which of the following is NOT a typical component of a token? a) Token type b) Token value c) Token length d) Token attributes Answer: c) Token length What is the primary function of “lexical tokens”? a) To represent basic units of source code for further processing b) To generate machine code c) To create syntax trees d) To perform semantic analysis Answer: a) To represent basic units of source code for further processing What is the role of a “token pattern” in lexical analysis? a) To define the structure and format of tokens b) To generate intermediate code c) To parse tokens into syntax trees d) To perform semantic analysis Answer: a) To define the structure and format of tokens Which of the following tools is used to generate lexical analyzers from regular expressions? a) Yacc b) Bison c) Lex d) Gcc Answer: c) Lex How does the lexical analyzer handle multi-character tokens? a) By using patterns that match sequences of characters b) By generating machine code c) By creating syntax trees d) By performing semantic analysis Answer: a) By using patterns that match sequences of characters What does the term “tokenizing” refer to? a) The process of converting source code into tokens b) The process of generating machine code c) The process of creating syntax trees d) The process of performing semantic analysis Answer: a) The process of converting source code into tokens What is the main function of a “lexical analyzer”? a) To break down source code into tokens b) To generate machine code c) To create syntax trees d) To perform semantic analysis Answer: a) To break down source code into tokens Which component in lexical analysis is responsible for recognizing patterns? a) Finite state machine b) Syntax analyzer c) Code generator d) Optimizer Answer: a) Finite state machine What does the lexical analyzer do when it encounters an invalid lexeme? a) Reports a lexical error b) Converts it into a valid token c) Generates machine code d) Creates a syntax tree Answer: a) Reports a lexical error Which of the following best describes a “token type”? a) The category of a token, such as keyword, identifier, or operator b) The actual sequence of characters in a lexeme c) The regular expression pattern used to define a token d) The machine code representation of a token Answer: a) The category of a token, such as keyword, identifier, or operator How does the lexical analyzer handle comments in the source code? a) It typically removes them and does not generate tokens for them b) It converts them into tokens c) It generates syntax errors for them d) It generates machine code for them Answer: a) It typically removes them and does not generate tokens for them What is the role of a “token stream” in the context of lexical analysis? a) It provides the sequence of tokens to subsequent stages of compilation b) It generates machine code c) It creates syntax trees d) It performs semantic analysis Answer: a) It provides the sequence of tokens to subsequent stages of compilation