6.7. Tokenisers

6.7. Tokenisers

Tokenise various languages.
Start python section to interscript/tokenisers/__init__.py[1]
     1: #line 4 "tokenizers.ipk"
     2: # tokenisers package
     3: 
End python section to interscript/tokenisers/__init__.py[1]


6.7.1. Python Tokeniser
6.7.2. Parsers