We use python's source code tokenizer to tokenize our TRG. Python's tokenizer returns several attributes for each token. We only extract the token type and the corresponding string attribute in form ...

From this snippet you can notice other 4 things: We put the data from the commlib in a np.array using the flag copy=False, this gives the wrapper the possibility to use the C++ data without having to ...