Update the TDL parsing API

[goodmami]

#81 made some parsing functions non-public (ones which weren't useful as part of the public API), but we left the tdl.lex() and tdl.tokenize() functions alone, even though they probably have limited utility outside of the tdl module. These functions can be deprecated if we adopt some changes to the API such that comments can be retrieved. Here is one proposal:

• drop tokenize() and its regex
• make lex() non-public (_lex()) and have it lex every token, not just the top-level constructions
• let parse() specify which entities it will yield (by default, for example, ('typedef', 'typeaddendum', 'instance')), such that 'comment' can be included and thus yielded

The parse() function thus behaves something like Python's ElementTree.iterparse()

For actual TDL parsing (with unification, etc.) and not just inspection, a load() function could take all the parsed entities (ignoring comments) and construct the type hierarchy and return some kind of compiled namespace object. There might be separate load_types() and load_instances() functions, like in the LKB, that restrict what kinds of entities can be parsed in a file.

[goodmami]

The underlying parsing logic will change dramatically, so maybe leave parse() as-is and deprecate it and add a new iterparse() function for the new logic. The load() functions can then make use of the new iterparse() function.