Lua Token READER patch

(NOT the Token Filter patch) 

Lua Token READER Patch adds a second source reader API to Lua 5.1.2, allowing C host programs to feed in a "token stream" instead of just raw Lua source code. The patch is backwards compatible (meaning: all Lua 5.1.2 functionality remains as is) but the split of LexState structure into token handling, and separate source parsing allows the parser to be removed from Lua core completely, in the future (if Lua authors would like, of course).

Unpatched Lua 5.1.2:

  1. C API uses 'lua_load' to load source code 
  2. Lua reads characters via 'lua_Reader' callback function (provided by host C program)
  3. Lua parses the characters into tokens (keywords, numbers, strings, control tokens s.a. + - * % . .. ...)
  4. Tokens are converted to LuaVM program code, and placed on Lua stack as a runnable function.

Unpatched Lua allows the resulting bytecode to be stored separately, and reused without the parsing, tokenizing and LuaVM conversion phases. There are two levels of input: source and precompiled bytecode.

With the Token READER patch, there is a third level of input between these two:

  1. C API uses 'lua_load_tokens' to initiate source loading.
  2. Lua reads tokens via 'lua_TokenReader' callback function (provided by host C program)
  3. The host C program parses source, on demand, into tokens.
  4. Lua converts to LuaVM program code, as usual.

I made this patch to allow luaSub to feed in tokens to the Lua core, without needing to translate them into string presentation (and then again back). Use of luaSuper together with this patch should give parsing speed on par with Lua 5.1.2 internal parser.

- test.c is a test program exercising the new  API

- README about the patch

- lua5.1.2-tokenreader-19102007.patch    (MD5: b79009c050fc0b446682a633fe6d8073)

 A possible future result of this approach is to cut the current Lua built-in parser away from the core, and more into the realm of the 'lua' command line front-end. This would be useful for scenarios where the core is not required to generate new code on the fly, and to slim down the core in general.

'Dostring' and others can still be implemented, with the front-end placing its functions under those names. It is up to the Lua authors to see, whether they would like to follow this path in future Lua releases.

- Asko Kauppi, 19-Oct-2007