I am trying to write propreties into a model and then query it.This part of mycode:
String directory = "EMAILADDRESS" ;
//create the dataset for the tdb store
Dataset ds = TDBFactory.createDataset(directory) ;
//create default rdf model
ds.begin(ReadWrite.WRITE);
Model model = ds.getDefaultModel() ;
//write to the tdb dataset
When I write this and then query the query shows no result ...but when I interchange the order of model and begin i.e.
Model model = ds.getDefaultModel() ;
//write to the tdb dataset
ds.begin(ReadWrite.WRITE);
Then it works fine!! but it sometimes gives this error:
com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Not in a transaction
I know that first way is correct but I don't understand why it doesn't respond to queries..This is code for quering:
public class test4query extends Object {
public static String[] arr=new String[30];
public void mai (String s) {
String directory = "EMAILADDRESS" ;
Dataset ds = TDBFactory.createDataset(directory) ;
ds.begin(ReadWrite.READ) ;
Model model = ds.getDefaultModel() ;
QueryExecution qExec = QueryExecutionFactory.create(s, ds) ;
int i=0;
try{
ResultSet rs = qExec.execSelect() ;
String x=rs.toString();
while (rs.hasNext()) {
QuerySolution qs = rs.next();
String rds;
if(qs.get("x")!=null) {
rds = qs.get("x").toString();
} else {
rds="hi";
}
if(rds==null) {
break;
}
System.out.println(rds);
arr[i] = rds;
i++;
}
} finally
{qExec.close() ;
ds.commit();
ds.end();
}
}
}
It is unclear when you get hat exception. The code example is full of parts that are commented out and does not use "m" at all.
You can not call ResultSetFormatter.out(rs) after you have called qExec.close or ds.commit.
Related
I am using the highlighting feature of Lucene to isolate matching terms for my query, but some of the matched terms are excessive.
I have some simple test cases which are delivered in an Ant project (download details below).
Materials
You can download the test case here: mydemo_with_libs.zip
That archive includes the Lucene 8.6.3 libraries which my test uses; if you prefer a copy without the JAR files you can download that from here: mydemo_without_libs.zip
The necessary libraries are: core, analyzers, queries, queryparser, highlighter, and memory.
You can run the test case by unzipping the archive into an empty directory and running the Ant command ant synsearch
Input
I have provided a short synonym list which is used for indexing and analysing in the highlighting methods:
cope,manage
jobs,tasks
simultaneously,at once
and there is one document being indexed:
Queues are a useful way of grouping jobs together in order to manage a number of them at once. You can:
hold or release multiple jobs at the same time;
group multiple tasks (for the same event);
control the priority of jobs in the queue;
Eventually log all events that take place in a queue.
Use either job.queue or task.queue in specifications.
Process
When building the index I am storing the text field, and using a custom analyzer. This is because (in the real world) the content I am indexing is technical documentation, so stripping out punctuation is inappropriate because so much of it may be significant in technical expressions. My analyzer uses a TechTokenFilter which breaks the stream up into tokens consisting of strings of words or digits, or individual characters which don't match the previous pattern.
Here's the relevant code for the analyzer:
public class MyAnalyzer extends Analyzer {
public MyAnalyzer(String synlist) {
if (synlist != "") {
this.synlist = synlist;
this.useSynonyms = true;
}
}
public MyAnalyzer() {
this.useSynonyms = false;
}
#Override
protected TokenStreamComponents createComponents(String fieldName) {
WhitespaceTokenizer src = new WhitespaceTokenizer();
TokenStream result = new TechTokenFilter(new LowerCaseFilter(src));
if (useSynonyms) {
result = new SynonymGraphFilter(result, getSynonyms(synlist), Boolean.TRUE);
result = new FlattenGraphFilter(result);
}
return new TokenStreamComponents(src, result);
}
and here's my filter:
public class TechTokenFilter extends TokenFilter {
private final CharTermAttribute termAttr;
private final PositionIncrementAttribute posIncAttr;
private final ArrayList<String> termStack;
private AttributeSource.State current;
private final TypeAttribute typeAttr;
public TechTokenFilter(TokenStream tokenStream) {
super(tokenStream);
termStack = new ArrayList<>();
termAttr = addAttribute(CharTermAttribute.class);
posIncAttr = addAttribute(PositionIncrementAttribute.class);
typeAttr = addAttribute(TypeAttribute.class);
}
#Override
public boolean incrementToken() throws IOException {
if (this.termStack.isEmpty() && input.incrementToken()) {
final String currentTerm = termAttr.toString();
final int bufferLen = termAttr.length();
if (bufferLen > 0) {
if (termStack.isEmpty()) {
termStack.addAll(Arrays.asList(techTokens(currentTerm)));
current = captureState();
}
}
}
if (!this.termStack.isEmpty()) {
String part = termStack.remove(0);
restoreState(current);
termAttr.setEmpty().append(part);
posIncAttr.setPositionIncrement(1);
return true;
} else {
return false;
}
}
public static String[] techTokens(String t) {
List<String> tokenlist = new ArrayList<String>();
String[] tokens;
StringBuilder next = new StringBuilder();
String token;
char minus = '-';
char underscore = '_';
char c, prec, subc;
// Boolean inWord = false;
for (int i = 0; i < t.length(); i++) {
prec = i > 0 ? t.charAt(i - 1) : 0;
c = t.charAt(i);
subc = i < (t.length() - 1) ? t.charAt(i + 1) : 0;
if (Character.isLetterOrDigit(c) || c == underscore) {
next.append(c);
// inWord = true;
}
else if (c == minus && Character.isLetterOrDigit(prec) && Character.isLetterOrDigit(subc)) {
next.append(c);
} else {
if (next.length() > 0) {
token = next.toString();
tokenlist.add(token);
next.setLength(0);
}
if (Character.isWhitespace(c)) {
// shouldn't be possible because the input stream has been tokenized on
// whitespace
} else {
tokenlist.add(String.valueOf(c));
}
// inWord = false;
}
}
if (next.length() > 0) {
token = next.toString();
tokenlist.add(token);
// next.setLength(0);
}
tokens = tokenlist.toArray(new String[0]);
return tokens;
}
}
Examining the index I can see that the index contains the separate terms I expect, including the synonym values. For example the text at the end of the first line has produced the terms
of
them
at , simultaneously
once
.
You
can
:
and the text at the end of the third line has produced the terms
same
event
)
;
When the application performs a search it analyzes the query without using the synonym list (because the synonyms are already in the index), but I have discovered that I need to include the synonym list when analyzing the stored text to identify the matching fragments.
Searches match the correct documents, but the code I have added to identify the matching terms over-performs. I won't show all the search method here, but will focus on the code which lists matched terms:
public static void doSearch(IndexReader reader, IndexSearcher searcher,
Query query, int max, String synList) throws IOException {
SimpleHTMLFormatter htmlFormatter = new SimpleHTMLFormatter("\001", "\002");
Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(query));
Analyzer analyzer;
if (synList != null) {
analyzer = new MyAnalyzer(synList);
} else {
analyzer = new MyAnalyzer();
}
// Collect all the docs
TopDocs results = searcher.search(query, max);
ScoreDoc[] hits = results.scoreDocs;
int numTotalHits = Math.toIntExact(results.totalHits.value);
System.out.println("\nQuery: " + query.toString());
System.out.println("Matches: " + numTotalHits);
// Collect matching terms
HashSet<String> matchedWords = new HashSet<String>();
int start = 0;
int end = Math.min(numTotalHits, max);
for (int i = start; i < end; i++) {
int id = hits[i].doc;
float score = hits[i].score;
Document doc = searcher.doc(id);
String docpath = doc.get("path");
String doctext = doc.get("text");
try {
TokenStream tokens = TokenSources.getTokenStream("text", null, doctext, analyzer, -1);
TextFragment[] frag = highlighter.getBestTextFragments(tokens, doctext, false, 100);
for (int j = 0; j < frag.length; j++) {
if ((frag[j] != null) && (frag[j].getScore() > 0)) {
String match = frag[j].toString();
addMatchedWord(matchedWords, match);
}
}
} catch (InvalidTokenOffsetsException e) {
System.err.println(e.getMessage());
}
System.out.println("matched file: " + docpath);
}
if (matchedWords.size() > 0) {
System.out.println("matched terms:");
for (String word : matchedWords) {
System.out.println(word);
}
}
}
Problem
While the correct documents are selected by these queries, and the fragments chosen for highlighting do contain the query terms, the highlighted pieces in some of the selected fragments extend over too much of the input.
For example, if the query is
+text:event +text:manage
(the first example in the test case) then I would expect to see 'event' and 'manage' in the highlighted list. But what I actually see is
event);
manage
Despite the highlighting process using an analyzer which breaks terms apart and treats punctuation characters as single terms, the highlight code is "hungry" and breaks on whitespace alone.
Similarly if the query is
+text:queeu~1
(my final test case) I would expect to only see 'queue' in the list. But I get
queue.
job.queue
task.queue
queue;
It is so nearly there... but I don't understand why the highlighted pieces are inconsistent with the index, and I don't think I should have to parse the list of matches through yet another filter to produce the correct list of matches.
I would really appreciate any pointers to what I am doing wrong or how I could improve my code to deliver exactly what I need.
Thanks for reading this far!
I managed to get this working by replacing the WhitespaceTokenizer and TechTokenFilter in my analyzer with a PatternTokenizer; the regular expression took a bit of work but once I had it all the matching terms were extracted with pinpoint accuracy.
The replacement analyzer:
public class MyAnalyzer extends Analyzer {
public MyAnalyzer(String synlist) {
if (synlist != "") {
this.synlist = synlist;
this.useSynonyms = true;
}
}
public MyAnalyzer() {
this.useSynonyms = false;
}
private static final String tokenRegex = "(([\\w]+-)*[\\w]+)|[^\\w\\s]";
#Override
protected TokenStreamComponents createComponents(String fieldName) {
PatternTokenizer src = new PatternTokenizer(Pattern.compile(tokenRegex), 0);
TokenStream result = new LowerCaseFilter(src);
if (useSynonyms) {
result = new SynonymGraphFilter(result, getSynonyms(synlist), Boolean.TRUE);
result = new FlattenGraphFilter(result);
}
return new TokenStreamComponents(src, result);
}
I am using script component in ssis with C# code using Microsoft.Analysisservices namespace to fetch the cube metadata. The code looks somewhat like this
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using Microsoft.AnalysisServices;
using System.Windows.Forms;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
//IDTSConnectionManager100 connMgr;
Server OLAPServer = new Server();
public override void AcquireConnections(object Transaction)
{
OLAPServer.Connect(this.Connections.OLAPConnection.ConnectionString);
}
public override void PreExecute()
{
base.PreExecute();
/*
Add your code here for preprocessing or remove if not needed
*/
}
public override void PostExecute()
{
base.PostExecute();
/*
Add your code here for postprocessing or remove if not needed
You can set read/write variables here, for example:
Variables.MyIntVar = 100
*/
}
public override void CreateNewOutputRows()
{
IDTSVariables100 vars = null;
string OLAPDBName;
VariableDispenser.LockOneForRead("OLAPDBName", ref vars);
Database OLAPDB;
OLAPDBName = vars[0].Value.ToString();
try
{
OLAPDB = OLAPServer.Databases.GetByName(OLAPDBName);
}
catch
{
return;
}
// loop through cubes
CubeCollection Cubes = OLAPDB.Cubes;
MeasureGroupCollection Mgroups;
CubeDimensionCollection Dimensions;
MeasureGroupDimensionCollection MgroupDims;
DimensionAttributeCollection Attributes;
foreach (Cube cb in Cubes)
{
//Test for one Measure Group
//MeasureGroup mgroup = Mgroups.GetByName("Inward Exposure");
Mgroups = cb.MeasureGroups;
// all dimensions associated with that Measure Group
// loop through Measure Groups
foreach (MeasureGroup mg in Mgroups)
{
// loop though all cube dimensions
Dimensions = cb.Dimensions;**strong text**
foreach (CubeDimension dim in Dimensions)
{
bool CanBeAnalysed = false;**strong text**
// loop through dimensions and see if dimension exists in mgroupDims (ie check if it can be analysed)
MgroupDims = mg.Dimensions;
foreach (MeasureGroupDimension mgd in MgroupDims)
{
if (mgd.CubeDimension == dim)
{
CanBeAnalysed = true;
break;
}
}
// loop through each Measure and Attribute a
String DimName = dim.Name;
bool DimVisible = dim.Visible;
String MgroupName = mg.Name;
String CubeName = cb.Name;
String MeasureExpression;
String Description;
// for every attribute in dimension
Attributes = dim.Dimension.Attributes;
foreach (DimensionAttribute Attr in Attributes)
{
String AttrName = Attr.Name;
bool AttrVisible = Attr.AttributeHierarchyVisible;
String AttrNameColumn = Attr.NameColumn.ToString();
String AttributeRelationship = Attr.AttributeRelationships.ToString();
// get every measure in measuregroup
foreach (Measure m in mg.Measures)
{
String MeasureName = m.Name.ToString();
bool MeasureVisible = m.Visible;
String MeasureNameColumn = m.Source.ToString();
if (m.MeasureExpression != null)
{
// MessageBox.Show(m.MeasureExpression.ToString());
MeasureExpression = m.MeasureExpression.ToString();
}
else
{
// MessageBox.Show(m.MeasureExpression.ToString());
MeasureExpression = " " ;
}
if (m.Description != null)
{
// MessageBox.Show(m.MeasureExpression.ToString());
Description = m.Description.ToString();
}
else
{
// MessageBox.Show(m.MeasureExpression.ToString());
Description = " ";
}
Output0Buffer.AddRow();
Output0Buffer.OLAPDBName = OLAPDBName;
Output0Buffer.CubeName = CubeName;
Output0Buffer.DimensionName = DimName;
Output0Buffer.DimensionVisible = DimVisible;
Output0Buffer.AttrDDSColumn = AttrNameColumn;
Output0Buffer.AttrName = AttrName;
Output0Buffer.AttrVisible = AttrVisible;
Output0Buffer.MeasureGroupName = MgroupName;
Output0Buffer.MeasureName = MeasureName;
Output0Buffer.MeasureVisible = MeasureVisible;
Output0Buffer.MeasureDDSColumn = MeasureNameColumn;
Output0Buffer.IsAnalysable = CanBeAnalysed;
Output0Buffer.MeasureExpression = MeasureExpression;
Output0Buffer.Description = Description;
Output0Buffer.AttributeRelationship = AttributeRelationship;
}
}
} // end of Cube Dim Loop
} // end of Measure Group loop
} // end of cube loop
}
}
I was successful in getting the cube metadata with the above code.However, i am stuck at getting the metadata of the perspective cube and the Relationships of the measure groups i.e whether the measure groups are many-many. Any help is very much appreciated.
Here is some code for detecting dimension relationships including many-to-many. See the GetDimensionUsage function:
https://raw.githubusercontent.com/BIDeveloperExtensions/bideveloperextensions/master/SSAS/PrinterFriendlyDimensionUsage.cs
Here is some code around navigating perspectives:
https://raw.githubusercontent.com/BIDeveloperExtensions/bideveloperextensions/master/SSAS/TriStatePerspectivesPlugin.cs
Start reading around the following line:
if (perspective.MeasureGroups.Contains(mg.Name))
I am trying to execute the query (Basic select statement with 10 fields). My table contains more than 500k rows. C# application returns the response with only 4260 rows. However Web UI returns all the records.
Why my code returns only partial data, What is the best way to select all the records and load into C# Data Table? If there is any code snippet it would be more helpful to me.
using Google.Apis.Auth.OAuth2;
using System.IO;
using System.Threading;
using Google.Apis.Bigquery.v2;
using Google.Apis.Bigquery.v2.Data;
using System.Data;
using Google.Apis.Services;
using System;
using System.Security.Cryptography.X509Certificates;
namespace GoogleBigQuery
{
public class Class1
{
private static void Main()
{
try
{
Console.WriteLine("Start Time: {0}", DateTime.Now.ToString());
String serviceAccountEmail = "SERVICE ACCOUNT EMAIL";
var certificate = new X509Certificate2(#"KeyFile.p12", "notasecret", X509KeyStorageFlags.Exportable);
ServiceAccountCredential credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(serviceAccountEmail)
{
Scopes = new[] { BigqueryService.Scope.Bigquery, BigqueryService.Scope.BigqueryInsertdata, BigqueryService.Scope.CloudPlatform, BigqueryService.Scope.DevstorageFullControl }
}.FromCertificate(certificate));
BigqueryService Service = new BigqueryService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "PROJECT NAME"
});
string query = "SELECT * FROM [publicdata:samples.shakespeare]";
JobsResource j = Service.Jobs;
QueryRequest qr = new QueryRequest();
string ProjectID = "PROJECT ID";
qr.Query = query;
qr.MaxResults = Int32.MaxValue;
qr.TimeoutMs = Int32.MaxValue;
DataTable DT = new DataTable();
int i = 0;
QueryResponse response = j.Query(qr, ProjectID).Execute();
string pageToken = null;
if (response.JobComplete == true)
{
if (response != null)
{
int colCount = response.Schema.Fields.Count;
if (DT == null)
DT = new DataTable();
if (DT.Columns.Count == 0)
{
foreach (var Column in response.Schema.Fields)
{
DT.Columns.Add(Column.Name);
}
}
pageToken = response.PageToken;
if (response.Rows != null)
{
foreach (TableRow row in response.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
while (true)
{
int StartIndexForQuery = DT.Rows.Count;
Google.Apis.Bigquery.v2.JobsResource.GetQueryResultsRequest SubQR = Service.Jobs.GetQueryResults(response.JobReference.ProjectId, response.JobReference.JobId);
SubQR.StartIndex = (ulong)StartIndexForQuery;
//SubQR.MaxResults = Int32.MaxValue;
GetQueryResultsResponse QueryResultResponse = SubQR.Execute();
if (QueryResultResponse != null)
{
if (QueryResultResponse.Rows != null)
{
foreach (TableRow row in QueryResultResponse.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
if (null == QueryResultResponse.PageToken)
{
break;
}
}
else
{
break;
}
}
}
else
{
Console.WriteLine("Response is null");
}
}
int TotalCount = 0;
if (DT != null && DT.Rows.Count > 0)
{
TotalCount = DT.Rows.Count;
}
else
{
TotalCount = 0;
}
Console.WriteLine("End Time: {0}", DateTime.Now.ToString());
Console.WriteLine("No. of records readed from google bigquery service: " + TotalCount.ToString());
}
catch (Exception e)
{
Console.WriteLine("Error Occurred: " + e.Message);
}
Console.ReadLine();
}
}
}
In this Sample Query get the results from public data set, In table contains 164656 rows but response returns 85000 rows only for the first time, then query again to get the second set of results. (But not known this is the only solution to get all the results).
In this sample contains only 4 fields, even-though it does not return all rows, in my case table contains more than 15 fields, I get response of ~4000 rows out of ~10k rows, I need to query again and again to get the remaining results for selecting 1000 rows takes time up to 2 minutes in my methodology so I am expecting best way to select all the records within single response.
Answer from User #:Pentium10
There is no way to run a query and select a large response in a single shot. You can either paginate the results, or if you can create a job to export to files, then use the files generated in your app. Exporting is free.
Step to run a large query and export results to files stored on GCS:
1) Set allowLargeResults to true in your job configuration. You must also specify a destination table with the allowLargeResults flag.
Example:
"configuration":
{
"query":
{
"allowLargeResults": true,
"query": "select uid from [project:dataset.table]"
"destinationTable": [project:dataset.table]
}
}
2) Now your data is in a destination table you set. You need to create a new job, and set the export property to be able to export the table to file(s). Exporting is free, but you need to have Google Cloud Storage activated to put the resulting files there.
3) In the end you download your large files from GCS.
It my turn to design the solution for better results.
Hoping this might help someone. One could retrieve next set of paginated result using PageToken. Here is the sample code for how to use PageToken. Although, I liked the idea of exporting for free. Here, I write rows to flat file but you could add them to your DataTable. Obviously, it is a bad idea to keep large DataTable in memory though.
public void ExecuteSQL(BigqueryService bqservice, String ProjectID)
{
string sSql = "SELECT r.Dealname, r.poolnumber, r.loanid FROM [MBS_Dataset.tblRemitData] R left join each [MBS_Dataset.tblOrigData] o on R.Dealname = o.Dealname and R.Poolnumber = o.Poolnumber and R.LoanID = o.LoanID Order by o.Dealname, o.poolnumber, o.loanid limit 100000";
QueryRequest _r = new QueryRequest();
_r.Query = sSql;
QueryResponse _qr = bqservice.Jobs.Query(_r, ProjectID).Execute();
string pageToken = null;
if (_qr.JobComplete != true)
{
//job not finished yet! expecting more data
while (true)
{
var resultReq = bqservice.Jobs.GetQueryResults(_qr.JobReference.ProjectId, _qr.JobReference.JobId);
resultReq.PageToken = pageToken;
var result = resultReq.Execute();
if (result.JobComplete == true)
{
WriteRows(result.Rows, result.Schema.Fields);
pageToken = result.PageToken;
if (pageToken == null)
break;
}
}
}
else
{
List<string> _fieldNames = _qr.Schema.Fields.ToList().Select(x => x.Name).ToList();
WriteRows(_qr.Rows, _qr.Schema.Fields);
}
}
The Web UI automatically flattens the data. This means that you see multiple rows for each nested field.
When you run the same query via the API, it won't be flattened, and you get fewer rows, as the nested fields are returned as objects. You should check if this is the case at you.
The other is that indeed you need to paginate through the results. Paging through list results has this explained.
If you want to do only one job, than you should write your query ouput to a table, than export the table as JSON, and download the export from GCS.
This question might sound lame but I am really confused...This is my code for normal query :
public class test4query extends Object {
public static String[] arr=new String[30];
public void mai (String s) {
String directory = "EMAILADDRESS" ;
Dataset ds = TDBFactory.createDataset(directory) ;
ds.begin(ReadWrite.READ) ;
Model model = ds.getDefaultModel() ;
QueryExecution qExec = QueryExecutionFactory.create(s, ds) ;
int i=0;
try{
ResultSet rs = qExec.execSelect() ;
String x=rs.toString();
while (rs.hasNext()) {
QuerySolution qs = rs.next();
String rds;
if(qs.get("x")!=null) {
rds = qs.get("x").toString();
} else {
rds="hi";
}
if(rds==null) {
break;
}
System.out.println(rds);
arr[i] = rds;
i++;
}
} finally
{qExec.close() ;
ds.commit();
ds.end();
}
}
}
But this does not work for delete queries ..It shows error :
Was expecting one of:
"base" ...
"prefix" ...
"select" ...
"describe" ...
"construct" ...
"ask" ...
I know some changes need to be made for update queries?Can somebody give some hint?Any link will be helpful!!
SPARQL Query and SPARQL Update are different languages, and there are different factories for parsing them. QueryFactory is for the SPARQL 1.1 Query Language. For the SPARQL 1.1 Update, you need to use UpdateFactory.
I took a dummy language for example:
It simply accepts one or more '!'.
its lexer and grammar rules are:
grammar Ns;
options {
output=AST;
ASTLabelType=CommonTree;
}
tokens {
NOTS;
}
#header {
package test;
}
#lexer::header {
package test;
}
ns : NOT+ EOF -> ^(NOTS NOT+);
NOT : '!';
ok, as you can see, this represents a language which accept '!' or '!!!' or '!!!!!'...
and I defined some meaningful classes to build ASTs:
public class Not {
public static final Not SINGLETON = new Not();
private Not() {
}
}
public class Ns {
private List<Not> nots;
public Ns(String nots) {
this.nots = new ArrayList<Not>();
for (int i = 0; i < nots.length(); i++) {
this.nots.add(Not.SINGLETON);
}
}
public String toString() {
String ret = "";
for (int i = 0; i < this.nots.size(); i++) {
ret += "!";
}
return ret;
}
}
and here's the tree grammar:
tree grammar NsTreeWalker;
options {
output = AST;
tokenVocab = Ns;
ASTLabelType = CommonTree;
}
#header {
package test;
}
ns returns [Ns ret] : ^(NOTS n=NOT+) {$ret = new Ns($n.text);};
and the main class code with some sample data to test the generated classes:
public class Test {
public static void main(String[] args) throws Exception {
ANTLRInputStream input = new ANTLRInputStream(new ByteArrayInputStream("!!!".getBytes("utf-8")));
NsLexer lexer = new NsLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
NsParser parser = new NsParser(tokens);
CommonTree root = (CommonTree) parser.ns().getTree();
NsTreeWalker walker = new NsTreeWalker(new CommonTreeNodeStream(root));
try {
NsTreeWalker.ns_return r = walker.ns();
System.out.println(r.ret);
} catch (RecognitionException e) {
e.printStackTrace();
}
}
}
but the final output printed is '!', other than the expecting '!!!'.
that's mainly because this line of code :
ns returns [Ns ret] : ^(NOTS n=NOT+) {$ret = new Ns($n.text);};
the $n above captured only one '!', I don't know how to capture all three tokens of '!', in other words , a list of '!' with $n.
Is there some one could help?thanks!
The fact that only one ! gets printed is because your rule:
ns returns [Ns ret]
: ^(NOTS n=NOT+) {$ret = new Ns($n.text);}
;
gets more or less translated as:
Token n = null
LOOP
n = match NOT_token
END
return new Ns(n.text)
Therefor, n.text will always be just a single !.
What you need to do is collect these NOT tokens in a list. In ANTLR you can create a list of tokens using the += operator instead of the "single token" operator =. So change your ns rule into:
ns returns [Ns ret]
: ^(NOTS n+=NOT+) {$ret = new Ns($n);}
;
which gets translated as:
List n = null
LOOP
n.add(match NOT_token)
END
return new Ns(n)
Be sure to change the constructor of your Ns class to take a List instead:
public Ns(List nots) {
this.nots = new ArrayList<Not>();
for (Object o : nots) {
this.nots.add(Not.SINGLETON);
}
}
after which the output of your test class would be:
!!!
Good luck!