How to rewrite token stream more than once using ANTLR4 - antlr

Question

I implement simple preprocessor using the great ANTLR4 library. The program itself runs in several iterations - in each iteration the future output is modified slightly.
Currently I use TokenStreamRewriter and its methods delete, insertAfter, replace and getText.
Unfortunately I can't manage to rewrite tokens that was rewritten before (got IllegalArgumentException). This is not a bug but according to the source code multiple replacement can't be achieved in any way.
I suppose that a proper solution exists as this appears to be a common problem. Could anyone please hint me? I'd rather use some existing and tested solution than reimplement the rewriter itself.
Maybe the rewriter isn't the right tool to use.
Thanks for help

Answer 1

Good evening
Now a dynamic code for the same problem. First you must have made visible in your listener class the Token stream and the rewriter
Here is the code of the constructor of my VB6Mylistener class
class VB6MYListener : public VB6ParserListener {
public: string FicName;
int numero ; // numero de la regle
wstring BaseFile;
CommonTokenStream* TOK ;
TokenStreamRewriter* Rewriter ;
// Fonctions pour la traduction avec le listener void functions
created by ANTLR4 ( contextes )
VB6MYListener( CommonTokenStream* tok , wstring baseFile, TokenStreamRewriter* rewriter , string Name)
{
TOK = tok; // Flux de tokens
BaseFile = baseFile; // Fichier entree en VB6
Rewriter = rewriter;
FicName = Name; // Nom du fichier courant pour suivi
}
Here in a context i cross with the listener. The Tokenstream is TOK visible by all the functions void
std::string retourchaine;
std::vector<std::string> TAB{};
for (int i = ctx->start->getTokenIndex(); i <= ctx->stop>getTokenIndex(); i++)
{
TAB.push_back(TOK->get(i)->getText()); // HERE TOK
}
for (auto element : TAB)
{
if (element == "=") { element = ":="; }
if (element != "As" && element != "Private" && element != "Public")
{
std::cout << element << std::endl;
retourchaine += element ; // retour de la chaine au contexte
}
}
retourchaine = retourchaine + " ;";
Rewriter->replace(ctx->start, ctx->stop, retourchaine );
`

Answer 2

A workaround I am using because I need to make a replacement in the token and the Tokenrewriter does not make the job correctly when you have multiple replacements in one context.
In each context I can make the stream of tokens visible and I use an array to copy all the tokens in the context and create a string with the replacement and after that, I use Rewriter->replace( ctx->start , ctx->stop , tokentext ) ;
Some code here for a context:
string TAB[265];
string tokentext = "";
for (int i = ctx->start->getTokenIndex(); i <= ctx->stop->getTokenIndex(); i++)
{
TAB[i] = TOK->get(i)->getText();
// if (TOK->get(i)->getText() != "As" && TOK->get(i)->getText() != "Private" && TOK->get(i)->getText() != "Public")
//if (TOK->get(i)->getText() == "=")
//{
if (TAB[i] == "=") { TAB[i] = ":="; }
// if (TAB[i] == "=") { TAB[i] = "="; } // autres changements
if (TAB[i] != "As" && TAB[i] != "Private" && TAB[i] != "Public") { tokentext += TAB[i]; }
cout << "nombre de tokens du contexte" << endl;
cout << i << endl;
}
tokentext = tokentext + " ;";
cout << tokentext << endl;
Rewriter->replace(ctx->start, ctx->stop, tokentext);
It's a a basic code I use to make the job robust. Hope this will be useful.

Answer 3

i think that rewriting the token stream is not a good idea, because you can't
treat the general case of a tree. The TokenStreamRewriter tool of ANTLR is usefulness. If you use a listener , you can't change the AST tree and the contexts created by ANTLR. you must use a Bufferedwriter to do the job for rewriting the context you change locally in the final file of your translation.
Thanks to Ewa Hechsman and her program on github on a transpiler from Pascal to Python.
I think it'a real solution for a professional project.
So i agree with Ira Baxter. we need a rewriting tree