Say I want to parse substrings with ParseKit, like the prefix of a word. So for example I want to parse 'preview' and 'review'. So my grammar might be:
#start = prefix 'view';
prefix = 'pre' | 're';
Now without modifying ParseKit I can match 'pre view' and 're view' but not 'preview' or 'review'. From looking at the documentation I guess I need to customize PKTokeinzer's word state because it is looking for whitespace to terminate a 'Word' token. How do I get around that?
Developer of ParseKit here.
I'm not sure I fully understand the question, but I think it sounds somewhat misguided.
If you are looking for a way to match sub-tokens or characters, Regular Expressions might be a better fit for your needs than ParseKit.
A ParseKit grammar matches against tokens produced by the ParseKit tokenizer (PKTokenizer class). Not individual characters.
It's not that it is impossible for PKTokenizer to produce a pre and and a view token from input of preview. But it would require customization of the code that I would call unwise and unnecessarily complicated. I think that is a bad idea.
If you want to use ParseKit (rather than Regex) anyway, you can simply do the sub-parsing in your assembler callbacks (instead of in the grammar).
So in the Grammar:
#start = either;
either = 'preview' | 'review';
And in ObjC:
- (void)parser:(PKParser *)p didMatchEither:(PKAssembly *)a {
PKToken *tok = [a pop];
NSString *str = tok.stringValue;
if ([str hasPrefix:#"pre"]) {
... // handle 'preview'
} else {
... // handle 'review'
}
}
Also remember that ParseKit Grammars support matching tokens via RegEx. So if you want to match all words that end in view:
#start = anyView;
anyView = /\b\w*?view\b/;
Hope that helps.
Related
I am very intrigued by the ability to add actions to ParseKit grammars. There is surprisingly little documentation on what is available in those actions. Say I have two rules like:
databaseName = Word;
createTableStmt ='CREATE' ('TEMP'| 'TEMPORARY')? 'TABLE' 'IF NOT EXISTS'? databaseName;
This obviously isn't a whole grammar but will serve as an example. When parsing i'd like to "return" a CreateTableStmt object that has certain properties. If I understand the tool correctly i'd add an action to the rule, do stuff then push it on the assembly which will carry it around for the next rule to deal with or use.
So for example it would look like:
createTableStmt ='CREATE' ('TEMP'| 'TEMPORARY')? 'TABLE' 'IF NOT EXISTS'? databaseName;
{
AnotherObj* dbName = Pop(); //gives me the top most object
CreateTableStmt* createTable = [[CreateTableStmt alloc] initWith:dbName];
//set if it was temporary
// set 'IF NOT EXISTS'
PUSH(createTable);//push back on stack for next rule to use
}
Then when everything is parsed I can just get that root object off the stack and it is a fully instantiated custom representation of the grammar. Somewhat like building an AST if i remember correctly. I can then do stuff with that representation much easier than with the passed in string.
My question is how can I see if it matched ('TEMP' | 'TEMPORARY') so I can set the value. Are those tokens on the stack? Is there a better way than to pop back to the 'CREATE' and see if we passed it. Should I be popping back to the bottom of the stack anyway on each match?
Also if my rule was instead
qualifiedTableName = (databaseName '.')? tableName (('INDEXED' 'BY' indexName) | ('NOT' 'INDEXED'))?;
Is it correct to assume that the action would not be called until the rule had been matched? So in this case when the action is called to the stack could look like:
possibly:
|'INDEXED'
|'NOT'
or:
|indexName (A custom object possibly)
|'BY'
|'INDEXED
|tableName (for sure will be here)
and possibly these
|'.' (if this is here I know the database name must be here) if not push last one on?
|databaseName
--------------(perhaps more things from other rules)
Are these correct assessments? Is there any other documentation on actions? I know it is heavily based on Antlr but its the subtle differences that can really get you in trouble.
Creator of ParseKit here. A few items:
ParseKit deprecation:
Just this week, I have forked ParseKit to a cleaner/smaller/faster library called PEGKit. ParseKit should be considered deprecated, and PEGKit should be used for all new development. Please move to PEGKit.
PEGKit is nearly identical to the grammar and code-gen features of ParseKit, and your ParseKit grammars are usable with PEGKit with a few small changes. In fact, all of the examples in your question here are usable with no changes in PEGKit.
See the Deprecation Notice in the ParseKit README.
And this tutorial on PEGKit.
Syntax errors in your grammar:
I spot 3 syntax errors in your grammar samples above (this applies equally to both ParseKit and PEGKit).
This line:
createTableStmt ='CREATE' ('TEMP'| 'TEMPORARY')? 'TABLE' 'IF NOT EXISTS'? databaseName;
Should be:
createTableStmt ='CREATE' ('TEMP'| 'TEMPORARY')? 'TABLE' ('IF' 'NOT' 'EXISTS')? databaseName;
Notice the break up of the invalid 'IF NOT EXISTS' construct into individual literal tokens. This is not only necessary, but also desireable so that variable whitespace between the words is allowed.
The POP() macro should be all upper case.
Your createTableStmt rule is missing a semicolon at the very end (after the action's closing }).
Before Answering:
Make sure you are using v0.3.1 PEGKit or later (HEAD of master). I fixed an important bug while finding the answer to your question, and my solutions below require this fix.
Answer to your first question:
My question is how can I see if it matched ('TEMP' | 'TEMPORARY') so I can set the value?
Good question! You basically have the right idea in your further comments above.
Specficially, I would probably break up the createTableStmt rule into 4 rules like this:
createTableStmt = 'CREATE'! tempOpt 'TABLE'! existsOpt databaseName ';'!;
databaseName = QuotedString;
tempOpt
= ('TEMP'! | 'TEMPORARY'!)
| Empty
;
existsOpt
= ('IF'! 'NOT'! 'EXISTS'!)
| Empty
;
Notice all of the vital ! discard directives for discarding unneeded literal tokens.
Also Notice that I've changed the last two rules to use | Empty rather than ?. This is so I can add Actions to the Empty alternatives (you'll see that in a sec).
Then you can either add Actions to your grammar, or use ObjC parser delegate callbacks if you prefer to work in pure code.
If you use Actions in your grammar, something like the following will work:
createTableStmt = 'CREATE'! tempOpt 'TABLE'! existsOpt databaseName ';'!
{
NSString *dbName = POP();
BOOL ifNotExists = POP_BOOL();
BOOL isTemp = POP_BOOL();
NSLog(#"create table: %#, %d, %d", dbName, ifNotExists, isTemp);
// go to town
// myCreateTable(dbName, ifNotExists, isTemp);
};
databaseName = QuotedString
{
// pop the string value of the `PKToken` on the top of the stack
NSString *dbName = POP_STR();
// trim quotes
dbName = [dbName substringWithRange:NSMakeRange(1, [dbName length]-2)];
// leave it on the stack for later
PUSH(dbName);
};
tempOpt
= ('TEMP'! | 'TEMPORARY'!) { PUSH(#YES); }
| Empty { PUSH(#NO); }
;
existsOpt
= ('IF'! 'NOT'! 'EXISTS'!) { PUSH(#YES); }
| Empty { PUSH(#NO); }
;
I've added this Grammar and a test case to the PEGKit project.
As for your second question, please break it out as a new SO question, and tag it ParseKit and PEGKit and I will get to it ASAP.
I just started using ParseKit to explore language creation and perhaps build a small toy DSL. However, the current SVN trunk from Google is throwing a -[PKToken intValue]: unrecognized selector sent to instance ... when parsing this grammar:
#start = identifier ;
identifier = (Letter | '_') | (letterOrDigit | '_') ;
letterOrDigit = Letter | Digit ;
Against this input:
foo
Clearly, I am missing something or have incorrectly configured my project. What can I do to fix this issue?
Developer of ParseKit here.
First, see the ParseKit Tokenization docs.
Basically, ParseKit can work in one of two modes: Let's call them Tokens Mode and Chars Mode. (There are no formal names for these two modes, but perhaps there should be.)
Tokens Mode is more popular by far. Virtually every example you will find of using ParseKit will show how to use Tokens Mode. I believe all of the documentation on http://parsekit.com is using Tokens Mode. ParseKit's grammar feature (that you are using in your example only works in Tokens Mode).
Chars Mode is a very little-known feature of ParseKit. I've never had anyone ask about it before.
So the differences in the modes are:
In Tokens Mode, the ParseKit Tokenizer emits multi-char tokens (like Words, Symbols, Numbers, QuotedStrings etc) which are then parsed by the ParseKit parsers you create (programmatically or via grammars).
In Chars Mode, the ParseKit Tokenizer always emits single-char tokens which are then parsed by the ParseKit parsers you create programmatically. (grammars don't currently work with this mode as this mode is not popular).
You could use Chars Mode to implement Regular Expresions which parse on a char-by-char basis.
For your example, you should be ignoring Chars Mode and just use Tokens Mode. The following Built-in Productions are for Chars Mode only. Do not use them in your grammars:
(PK)Letter
(PK)Digit
(PK)Char
(PK)SpecificChar
Notice how all of those Productions sound like they match individual chars. That's because they do.
Your example above should probably look like:
#start = identifier;
identifier = Word; // by default Words start with a-zA-Z_ and contain -0-9a-zAZ_'
Keep in mind the Productions in your grammars (parsers like identifier) will be working on Tokens already emitted from ParseKit's Tokenizer. Not individual chars.
IOW: by the time your grammar goes to work parsing input, the input has already been tokenized into Tokens of type Word, Number, Symbol, QuotedString, etc.
Here are all of the Built-in Productions available for use in your Grammar:
Word
Number
Symbol
QuotedString
Comment
Any
S // Whitespace. only available when #preservesWhitespaceTokens=YES. NO by default.
Also:
DelimitedString('start', 'end', 'allowedCharset')
/xxx/i // RegEx match
There are also operators for composite parsers:
// Sequence
| // Alternation
? // Optional
+ // Multiple
* // Repetition
~ // Negation
& // Intersection
- // Difference
I've a fairly simple question about ParseKit and parsing timestamps... how do I go about forcing the symbolic-nature of a dot/period.
For example, if I am trying to parse 2008-01-25, I could use something like date = /\d{4}/ '-' /\d{2}/ '-' /\d{2}/. In fact, there is a date.grammar shipped with ParseKit that does exactly this (interestingly enough, though, the provided grammar doesn't work in the DemoApp unless you add #symbolState='-';, but I digress...)
However, what do I do if I want to parse a date with dots in it... for example, 2008.01.25 or 2008-01-25-12.34.45. I've tried added '.' to the #symbolState directive but it just keeps getting ignored. Note that I am relying on the DemoApp to test my grammars at the moment... not sure if that makes any difference.
Any thoughts would be much appreciated.
Developer of ParseKit here.
First, thanks for the heads up on the bug in the date.grammar file. I have fixed it.
As for your main question, I'm pretty sure what you are trying was not possible with ParseKit until now.
That is, ParseKit's tokenizer (PKTokenizer) was not able to produce only whole number Number tokens. Numbers were always tokenized as floating point which means it was impossible to parse input like 3.14 as three separate tokens 3 (Number) . (Symbol) 14 (Number). Rather it would always be tokenized as 3.14.
Good news: I've added this capability with a new method:
-[PKNumberState allowsFloatingPoint]
which defaults to YES.
And I added a matching Tokenizer Directive which you can use in your ParseKit Grammars like:
#allowsFloatingPoint = NO;
NOTE you'll need to checkout the latest HEAD of trunk on Google Code to see this feature.
So, here's an example date grammar which does roughly what you were asking for with the new feature:
#symbolState = '.';
#allowsFloatingPoint = NO;
#start = date;
date = year dot month dot day;
year = /\d{4}/;
month = /\d{2}/;
day = /\d{2}/;
dot = '.';
I'm wondering if a simpler idea might be to get parseKit to simply parse the date as a string, and then hand it off to the NSDate::dateWithNaturalLanguageString:locale: orNSDate::dateWithNaturalLanguageString: for processing.
From Parsekit: how to match individual quote characters?
If you define a parser:
#start = int;
int = /[+-]?[0-9]+/
Unfortunately it isn't going to be parsing any integers prefixed with a "+", unless you include:
#numberState = "+" // at the top.
In the number parse above, the "Symbol" default parser wasn't even mentioned, yet it is still active and overrides user defined parsers.
Okay so with numbers you can still fix it by adding the directive. What if you're trying to create a parser for "++"? I haven't found any directive that can make the following parser work.
#start = plusplus;
plusplus = "++";
The effects of default parsers on the user parser seems so arbitrary. Why can't I parse "++"?
Is it possible to just turn off default Parsers altogether? They seem to get in the way if I'm not doing something common.
Or maybe I've got it all wrong.
EDIT:
I've found a parser that would parse plus plus:
#start = plusplus;
plusplus = plus plus;
plus = "+";
I am guessing the answer is: the literal symbols defined in your parser cannot overlap between default parsers; It must be contained completely by at least once of them.
Developer of ParseKit here.
I have a few responses.
I think you'll find the ParseKit API highly elegant and sensible, the more you learn. Keep in mind that I'm not tooting my own horn by saying that. Although I built ParseKit, I did not design the ParseKit API. Rather, the design of ParseKit is based almost entirely on the designs found in Steven Metsker's Building Parsers In Java. I highly recommend you checkout the book if you want to deeply understand ParseKit. Plus it's a fantastic book about parsing in general.
You're confusing Tokenizer States with Parsers. They are two distinct things, but the details are more complex than I can answer here. Again, I recommend Metsker's book.
In the course of answering your question, I did find a small bug in ParseKit. Thanks! However, it was not affecting your outcome described above as you were not using the correct grammar to get the outcome it seems you were looking for. You'll need to update your source code from The Google Code Project now, or else my advice below will not work for you.
Now to answer your question.
I think you are looking for a grammar which both recognizes ++ as a single multi-char Symbol token and also recognizes numbers with leading + chars as explicitly-positive numbers rather than a + Symbol token followed by a Number token.
The correct grammar I believe you are looking for is something like this:
#symbols = '++'; // declare ++ as a multi-char symbol
#numberState = '+'; // allow explicitly-positive numbers
#start = (Number|Symbol)*;
Input like this:
++ +1 -2 + 3 ++
Will be tokenized like so:
[++, +1, -2, +, 3, ++]++/+1/-2/+/3/++^
Two reminders:
Again, you will need to update your source code now to see this work correctly. I had to fix a bug in this case.
This stuff is tricky, and I recommend reading Metsker's book to fully understand how ParseKit works.
Hey guys, given a data set in plain text such as the following:
==Events==
* [[312]] – [[Constantine the Great]] is said to have received his famous [[Battle of Milvian Bridge#Vision of Constantine|Vision of the Cross]].
* [[710]] – [[Saracen]] invasion of [[Sardinia]].
* [[939]] – [[Edmund I of England|Edmund I]] succeeds [[Athelstan of England|Athelstan]] as [[King of England]].
*[[1275]] – Traditional founding of the city of [[Amsterdam]].
*[[1524]] – [[Italian Wars]]: The French troops lay siege to [[Pavia]].
*[[1553]] – Condemned as a [[Heresy|heretic]], [[Michael Servetus]] is [[burned at the stake]] just outside [[Geneva]].
*[[1644]] – [[Second Battle of Newbury]] in the [[English Civil War]].
*[[1682]] – [[Philadelphia]], [[Pennsylvania]] is founded.
I would like to end up with an NSDictionary or other form of collection so that I can have the year (The Number on the left) mapping to the excerpt (The text on the right). So this is what the 'template' is like:
*[[YEAR]] – THE_TEXT
Though I would like the excerpt to be plain text, that is, no wiki markup so no [[ sets. Actually, this could prove difficult with alias links such as [[Edmund I of England|Edmund I]].
I am not all that experienced with regular expressions so I have a few questions. Should I first try to 'beautify' the data? For example, removing the first line which will always be ==Events==, and removing the [[ and ]] occurrences?
Or perhaps a better solution: Should I do this in passes? So for example, the first pass I can separate each line into * [[710]] and [[Saracen]] invasion of [[Sardinia]]. and store them into different NSArrays.
Then go through the first NSArray of years and only get the text within the [[]] (I say text and not number because it can be 530 BC), so * [[710]] becomes 710.
And then for the excerpt NSArray, go through and if an [[some_article|alias]] is found, make it only be [[alias]] somehow, and then remove all of the [[ and ]] sets?
Is this possible? Should I use regular expressions? Are there any ideas you can come up with for regular expressions that might help?
Thanks! I really appreciate it.
EDIT: Sorry for the confusion, but I only want to parse the above data. Assume that that's the only type of markup that I will encounter. I'm not necessarily looking forward to parsing wiki markup in general, unless there is already a pre-existing library which does this. Thanks again!
This code assumes you are using RegexKitLite:
NSString *data = #"* [[312]] – [[Constantine the Great]] is said to have received his famous [[Battle of Milvian Bridge#Vision of Constantine|Vision of the Cross]].\n\
* [[710]] – [[Saracen]] invasion of [[Sardinia]].\n\
* [[939]] – [[Edmund I of England|Edmund I]] succeeds [[Athelstan of England|Athelstan]] as [[King of England]].\n\
*[[1275]] – Traditional founding of the city of [[Amsterdam]].";
NSString *captureRegex = #"(?i)(?:\\* *\\[\\[)([0-9]*)(?:\\]\\] \\– )(.*)";
NSRange captureRange;
NSRange stringRange;
stringRange.location = 0;
stringRange.length = data.length;
do
{
captureRange = [data rangeOfRegex:captureRegex inRange:stringRange];
if ( captureRange.location != NSNotFound )
{
NSString *year = [data stringByMatching:captureRegex options:RKLNoOptions inRange:stringRange capture:1 error:NULL];
NSString *textStuff = [data stringByMatching:captureRegex options:RKLNoOptions inRange:stringRange capture:2 error:NULL];
stringRange.location = captureRange.location + captureRange.length;
stringRange.length = data.length - stringRange.location;
NSLog(#"Year:%#, Stuff:%#", year, textStuff);
}
}
while ( captureRange.location != NSNotFound );
Note that you really need to study up on RegEx's to build these well, but here's what the one I have is saying:
(?i)
Ignore case, I could have left that out since I'm not matching letters.
(?:\* *\[\[)
?: means don't capture this block, I escape * to match it, then there are zero or more spaces (" *") then I escape out two brackets (since brackets are also special characters in a regex).
([0-9]*)
Grab anything that is a number.
(?:\]\] \– )
Here's where we ignore stuff again, basically matching " – ". Note any "\" in the regex, I have to add another one to in the Objective-C string above since "\" is a special character in a string... and yes that means matching a regex escaped single "\" ends up as "\\" in an Obj-C string.
(.*)
Just grab anything else, by default the RegEX engine will stop matching at the end of a line which is why it doesn't just match everything else. You'll have to add code to strip out the [[LINK]] stuff from the text.
The NSRange variables are used to keep matching through the file without re-matching original matches. So to speak.
Don't forget after you add the RegExKitLite class files, you also need to add the special linker flag or you'll get lots of link errors (the RegexKitLite site has installation instructions).
I'm no good with regular expressions, but this sounds like a job for them. I imagine a regex would sort this out for you quite easily.
Have a look at the RegexKitLite library.
If you want to be able to parse Wikitext in general, you have a lot of work to do. Just one complicating factor is templates. How much effort do you want to go to cope with these?
If you're serious about this, you probably should be looking for an existing library which parses Wikitext. A brief look round finds this CPAN library, but I have not used it, so I can't cite it as a personal recommendation.
Alternatively, you might want to take a simpler approach and decide which particular parts of Wikitext you're going to cope with. This might be, for example, links and headings, but not lists. Then you have to focus on each of these and turn the Wikitext into whatever you want that to look like. Yes, regular expressions will help a lot with this bit, so read up on them, and if you have specific problems, come back and ask.
Good luck!