4.6.2 Finding Tokens by String Literals

If the grammar uses literal string tokens, there are two ways that yylex can determine the token type codes for them:

• If the grammar defines symbolic token names as aliases for the literal string tokens, yylex can use these symbolic names like all others. In this case, the use of the literal string tokens in the grammar file has no effect on yylex.

  This is the preferred approach.

• yylex can search for the multicharacter token in the yytname table. This method is discouraged: the primary purpose of string aliases is forging good error messages, not describing the spelling of keywords. In addition, looking for the token type at runtime incurs a (small but noticeable) cost.

  The index of the token in the table is the token type’s code. The name of a multicharacter token is recorded in yytname with a double-quote, the token’s characters, and another double-quote. The token’s characters are escaped as necessary to be suitable as input to Bison.

  Here’s code for looking up a multicharacter token in yytname, assuming that the characters of the token are stored in token_buffer, and assuming that the token does not contain any characters like ‘"’ that require escaping.

  for (int i = 0; i < YYNTOKENS; i++)
    {
      if (yytname[i]
          && yytname[i][0] == '"'
          && ! strncmp (yytname[i] + 1, token_buffer,
                        strlen (token_buffer))
          && yytname[i][strlen (token_buffer) + 1] == '"'
          && yytname[i][strlen (token_buffer) + 2] == 0)
        break;
    }
  

  The yytname table is generated only if you use the %token-table declaration. See Decl Summary.