I have generated a JSON parser with ANTLR. To parse a JSON file, I must call the first rule in the JSON grammar:
parser = new JSONParser(...);
parser.json();
If I only want to parse a JSON array, can I skip all the JSON tokens up until the first token that starts an array and then call:
parser.array();
My guess is it won't parse until the end of the file, but instead it will stop at the end of the array in the JSON file. That's ok. But I'm not sure if it is allowed to call array() without any context, such as the correct JSON lexer mode.
Is there an example that describes how to do this?
You could call seek() on your TokenStream before initiating the parse. (I haven't directly tested it, but that should advance you TokenStream to the token where you wish to begin parsing).
Your intuition regarding calling the parser.array() method, and that it will parse until it has completed parsing the array that starts at your position in the token stream, is correct. It will not parse to the end of the input.
If you don't know the index to seek to, perhaps you want to locate the first array, to start there, just call write a loop that calls LA() to get the next token (it's short for "Look ahead"). If it's not the '[' token, then consume() it. If it is, then you're at the right position and can call parser.array() and will will parse just that array and return.
Following up to Mike's answer, the lexer works independently of the parser, and you can't (easily) backup the lexer. But, you can work with CommonTokenStream to position where you want to parse:
var lexer = new JSONLexer(str);
var tokens = new CommonTokenStream(lexer);
for (int i = 0; ; ++i) {
var t = tokens.LA(i+1);
var token = tokens.LT(i+1);
if (token.Text == "[")
{
tokens.Seek(i);
break;
}
}
var parser = new JSONParser(tokens);
var tree = parser.arr();
This code does a string comparison, but if you define a lexer symbol for '[', then you can break on the Type instead.
Related
Using C# and ANTLR4, I'm trying to parse a simple grammar, which is just a simple assign statement, which would look like:
int someinteger = 3;.
Below are my parser rules, which contain a compile unit, block and basic statement.
//The final compile unit sent to the interpreter.
compileUnit
: block EOF
;
//A block, array of statements.
block: statement*
;
//A single statement.
statement: stat_ass;
//An assign statement.
stat_ass: IDENTIFIER IDENTIFIER SET_EQUALS INTEGER ENDLINE;
When parsing int banana = 142;, the tokens returned are:
[IDENTIFIER, int]
[IDENTIFIER, banana]
[SET_EQUALS, =]
[INTEGER, 142]
[ENDLINE, ;]
However, when printing my parse tree, it just contains a block which has no statements.
ANTLR Parse Tree:
([] [10] <EOF>)
Can someone enlighten me on why this fails? Apologies if this is a simple mistake, I've run out of options I can think of to fix this.
Program.cs:
using Antlr4.Runtime;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace stork
{
class Program
{
static void Main(string[] args)
{
//Test input string.
string input = "int banana = 142;";
var chars = new AntlrInputStream(input);
var lexer = new storkLexer(chars);
var tokens = new CommonTokenStream(lexer);
//Debug print.
ANTLRDebug.PrintTokens(lexer);
//Debug print tree.
var parser = new storkParser(tokens);
ANTLRDebug.PrintParseList(parser);
//Getting tree.
parser.BuildParseTree = true;
var tree = parser.compileUnit();
}
}
}
ANTLRDebug.cs
https://github.com/c272/stork-lang/blob/master/stork/ANTLRDebug.cs
stork.g4
https://github.com/c272/stork-lang/blob/master/stork/Stork.g4
Your ANTLRDebug.PrintTokens method iterates over all the tokens from the lexer, consuming all of them. Afterwards the lexer is empty (it's like an iterator that way), so you're invoking the parser on an empty token stream.
You should call lexer.reset() after calling ANTLRDebug.PrintTokens (or call it at the end of that method) to reset the lexer to the beginning of the input stream.
PS: I recommend calling ToStringTree(parser) instead of just ToStringTree() as that will produce more readable output (rule names instead of numbers).
I have a lexer which puts every token the parser is interested in into the default channel and all comment-tokens in channel 1.
The default channel is used to create the actual tree while the comment channel is used to seperate the tokens and to store all comments.
Look at this scribble:
In chapter 12.1 p. 206-208 in The Definitive ANTLR4 Reference there is a comparable situation where comment tokens are shifted inside the token stream. The represented approach is to read out the comment-channel in an exit-method inside the parser.
In my opinion this is a very rough option for my problem, because i don't want to overwhelm my listener with that back-looking operations. Is there a possibility to override a method which puts tokens inside the comment-channel?
It looks like you misunderstand how channels work in ANTLR. What happens is that the lexer, as it comes along a token, assigns the default channel (just a number) during initialization of the token. That value is only changed when the lexer finds a -> channel() command or you change it explicitely in a lexer action. So there is nothing to do in a listener or whatever to filter out such tokens.
Later when you want to get all tokens "in" a given channel (i.e. all tokens that have a specific channel number assigned) you can just iterate over all tokens returned by your token stream and compare the channel value. Alternatively you can create a new CommonTokenStream instance and pass it the channel your are interested in. It will then only give you those tokens from that channel (it uses a token source, e.g. a lexer, to get the actual tokens and cache them).
I found out, that there is a easy way to override how tokens are created. To do this, one can override a method inside the CommonTokenFactory and give it to the Lexer. At this point i can check the channel and i am able to push the tokens in a separate set.
In my opinion this is a little bit hacky, but i do not need to iterate over the whole commonTokenStream later on.
This code is only to demonstrate the idea behind (in C#) .
internal class HeadAnalyzer
{
#region Methods
internal void AnalyzeHeader(Stream headerSourceStream)
{
var antlrFileStream =
new AntlrInputStream(headerSourceStream);
var mcrLexer = new MCRLexer(antlrFileStream);
var commentSaverTokenFactory = new MyTokenFactory();
mcrLexer.TokenFactory = commentSaverTokenFactory;
var commonTokenStream = new CommonTokenStream(mcrLexer);
var mcrParser = new MCRParser(commonTokenStream);
mcrParser.AddErrorListener(new DefaultErrorListener());
MCRParser.ProgramContext tree;
try
{
tree = mcrParser.program(); // create the tree
}
catch (SyntaxErrorException syntaxErrorException)
{
throw new NotImplementedException();
}
var headerContext = new HeaderContext();
var headListener = new HeadListener(headerContext);
ParseTreeWalker.Default.Walk(headListener, tree);
var comments = commentSaverTokenFactory.CommentTokens; // contains all comments :)
}
#endregion
}
internal class MyTokenFactory : CommonTokenFactory
{
internal readonly List<CommonToken> CommentTokens = new List<CommonToken>();
public override CommonToken Create(Tuple<ITokenSource, ICharStream> source, int type, string text, int channel, int start, int stop, int line, int charPositionInLine)
{
var token = base.Create(source, type, text, channel, start, stop, line, charPositionInLine);
if (token.Channel == 1)
{
CommentTokens.Add(token);
}
return token;
}
}
Maybe there are some better approaches. For my usecase it works as expected.
I am using Google diff-match-patch JAVA plugin to create patch between two JSON strings and storing the patch to database.
diff_match_patch dmp = new diff_match_patch();
LinkedList<Patch> diffs = dmp.patch_make(latestString, originalString);
String patch = dmp.patch_toText(diffs); // Store patch to DB
Now is there any way to use this patch to re-create the originalString by passing the latestString?
I google about this and found this very old comment # Google diff-match-patch Wiki saying,
Unpatching can be done by just looping through the diff, swapping
DIFF_INSERT with DIFF_DELETE, then applying the patch.
But i did not find any useful code that demonstrates this. How could i achieve this with my existing code ? Any pointers or code reference would be appreciated.
Edit:
The problem i am facing is, in the front-end i am showing a revisions module that shows all the transactions of a particular fragment (take for example an employee details), like which user has updated what details etc. Now i am recreating the fragment JSON by reverse applying each patch to get the current transaction data and show it as a table (using http://marianoguerra.github.io/json.human.js/). But some JSON data are not valid JSON and I am getting JSON.parse error.
I was looking to do something similar (in C#) and what is working for me with a relatively simple object is the patch_apply method. This use case seems somewhat missing from the documentation, so I'm answering here. Code is C# but the API is cross language:
static void Main(string[] args)
{
var dmp = new diff_match_patch();
string v1 = "My Json Object;
string v2 = "My Mutated Json Object"
var v2ToV1Patch = dmp.patch_make(v2, v1);
var v2ToV1PatchText = dmp.patch_toText(v2ToV1Patch); // Persist text to db
string v3 = "Latest version of JSON object;
var v3ToV2Patch = dmp.patch_make(v3, v2);
var v3ToV2PatchTxt = dmp.patch_toText(v3ToV2Patch); // Persist text to db
// Time to re-hydrate the objects
var altV3ToV2Patch = dmp.patch_fromText(v3ToV2PatchTxt);
var altV2 = dmp.patch_apply(altV3ToV2Patch, v3)[0].ToString(); // .get(0) in Java I think
var altV2ToV1Patch = dmp.patch_fromText(v2ToV1PatchText);
var altV1 = dmp.patch_apply(altV2ToV1Patch, altV2)[0].ToString();
}
I am attempting to retrofit this as an audit log, where previously the entire JSON object was saved. As the audited objects have become more complex the storage requirements have increased dramatically. I haven't yet applied this to the complex large objects, but it is possible to check if the patch was successful by checking the second object in the array returned by the patch_apply method. This is an array of boolean values, all of which should be true if the patch worked correctly. You could write some code to check this, which would help check if the object can be successfully re-hydrated from the JSON rather than just getting a parsing error. My prototype C# method looks like this:
private static bool ValidatePatch(object[] patchResult, out string patchedString)
{
patchedString = patchResult[0] as string;
var successArray = patchResult[1] as bool[];
foreach (var b in successArray)
{
if (!b)
return false;
}
return true;
}
I'm new to Javascript and the google-api and still struggling with casting and basic functionality.
I'm trying to copy a column of strings harvested from a sheet to another array for manipulation and running into various errors. So far I have tried the following ( that have not generated straight up syntax errors )...
// Character VALIDATION function
function validateCharacter(characterName) {
var sheetName = ("!" + characterName);
var characterSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
// characterData array will hold the values from the characters sheet
var dataRange=characterSheet.getDataRange();
var characterData = dataRange.getValues();
var Physicals;
Physicals.copyofrange(characterData,[14][0],[31][0]);
console.log(Physicals);
}
and
// Character VALIDATION function
function validateCharacter(characterName) {
var sheetName = ("!" + characterName);
var characterSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
// characterData array will hold the values from the characters sheet
var dataRange=characterSheet.getDataRange();
var characterData = dataRange.getValues();
var Physicals;
for (i=0;i<17;i++){
Physicals[i]=characterData[i+13][0];
}
console.log(Physicals);
}
I know that Java handles strings as pointers, but I'm not clear if Javascript does and I thought of the for loop first but I'm getting
TypeError: Cannot set property "0.0" of undefined to "Agile". (line 40).
when I do that. I'm assuming I'm missing something... dimensional? about the declaration of the array to be copied to but I can't figure out what. I've looked over various solutions involving Java declarations for arrays but none of those seem to apply or be allowed in the google-spreadsheets-api Javascript. It seemed to not even acknowledge the String declaration when I tried String[]=new String etc. Frankly I don't care how it's implemented just so it works. Efficiency isn't a big concern on this project. Thank you.
OK, figured it out. After seeing all the Java things that Javascript does not support, I found some documentation specifically on Javascript string handling and figured out I have to cast Physicals as an empty array then "push" the values into it in the loop rather than assigning them with = statements, as such.
// Character VALIDATION function
function validateCharacter(characterName) {
var sheetName = ("!" + characterName);
var characterSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
// characterData array will hold the values from the characters sheet
var dataRange=characterSheet.getDataRange();
var characterData = dataRange.getValues();
var Physicals=[];
for (i=0;i<17;i++){
Physicals.push(characterData[i+13][0]);
}
return(Physicals);
}
Knowing the distinction between Java, Javascript, and Guava actually helped a great deal since they seem to have wildly different function handling despite similar names and common ancestry.
I need to create a grammar for a language with forward references. I think that the easiest way to achieve this is to make several passes on the generated AST, but I need a way to store symbol information in the tree.
Right now my parser correctly generates an AST and computes scopes of the variables and function definitions. The problem is, I don't know how to save the scope information into the tree.
Fragment of my grammar:
composite_instruction
scope JScope;
#init {
$JScope::symbols = new ArrayList();
$JScope::name = "level "+ $JScope.size();
}
#after {
System.out.println("code block scope " +$JScope::name + " = " + $JScope::symbols);
}
: '{' instruction* '}' -> ^(INSTRUCTION_LIST instruction*)
;
I would like to put a reference to current scope into a tree, something like:
: '{' instruction* '}' -> ^(INSTRUCTION_LIST instruction* {$JScope::symbols})
Is it even possible? Is there any other way to store current scopes in a generated tree? I can generate the scope info in a tree grammar, but it won't change anything, because I still have to store it somewhere for the second pass on the tree.
To my knowledge, the syntax for the rewrite rules doesn't allows for directly assigning values as your tentative snippet suggests. This is in part due to the fact that the parser wouldn't really know to what part of the tree/node the values should be added to.
However, one of the cool features of ANTLR-produced ASTs is that the parser makes no assumptions about the type of the Nodes. One just needs to implement a TreeAdapator which serves as a factory for new nodes and as a navigator of the tree structure. One can therefore stuff whatever info may be needed in the nodes, as explained below.
ANTLR provides a default tree node implementation, CommonTree, and in most cases (as in the situation at hand) we merely need to
subclass CommonTree by adding some custom fields to it
subclass the CommonTreeAdaptor to override its create() method, i.e. the way it produces new nodes.
but one could also create a novel type of node altogher, for some odd graph structure or whatnot. For the case at hand, the following should be sufficient (adapt for the specific target language if this isn't java)
import org.antlr.runtime.tree.*;
import org.antlr.runtime.Token;
public class NodeWithScope extends CommonTree {
/* Just declare the extra fields for the node */
public ArrayList symbols;
public string name;
public object whatever_else;
public NodeWithScope (Token t) {
super(t);
}
}
/* TreeAdaptor: we just need to override create method */
class NodeWithScopeAdaptor extends CommonTreeAdaptor {
public Object create(Token standardPayload) {
return new NodeWithScope(standardPayload);
}
}
One then needs to slightly modify the way the parsing process is started, so that ANTLR (or rather the ANTLR-produced parser) knows to use the NodeWithScopeAdaptor rather than CommnTree.
(Step 4.1 below, the rest if rather standard ANTLR test rig)
// ***** Typical ANTLR pipe rig *****
// ** 1. input stream
ANTLRInputStream input = new ANTLRInputStream(my_input_file);
// ** 2, Lexer
MyGrammarLexer lexer = new MyGrammarLexer(input);
// ** 3. token stream produced by lexer
CommonTokenStream tokens = new CommonTokenStream(lexer);
// ** 4. Parser
MyGrammarParser parser = new MyGrammarParser(tokens);
// 4.1 !!! Specify the TreeAdapter
NodeWithScopeAdaptor adaptor = new NodeWithScopeAdaptor();
parser.setTreeAdaptor(adaptor); // use my adaptor
// ** 5. Start process by invoking the root rule
r = parser.MyTopRule();
// ** 6. AST tree
NodeWithScope t = (NodeWithScope)r.getTree();
// ** 7. etc. parse the tree or do whatever is needed on it.
Finally your grammar would have to be adapted with something akin to what follows
(note that the node [for the current rule] is only available in the #after section. It may however reference any token attribute and other contextual variable from the grammar-level, using the usual $rule.atrribute notation)
composite_instruction
scope JScope;
#init {
$JScope::symbols = new ArrayList();
$JScope::name = "level "+ $JScope.size();
}
#after {
($composite_instruction.tree).symbols = $JScope::symbols;
($composite_instruction.tree).name = $JScope::name;
($composite_instruction.tree).whatever_else
= new myFancyObject($x.Text, $y.line, whatever, blah);
}
: '{' instruction* '}' -> ^(INSTRUCTION_LIST instruction*)
;