Is there any function that does what NSLog does but without the new line at the end?
see this http://borkware.com/quickies/one?topic=NSString
excerpted from that page:
void LogIt (NSString *format, ...)
{
va_list args;
va_start (args, format);
NSString *string;
string = [[NSString alloc] initWithFormat: format arguments: args];
va_end (args);
printf ("%s\n", [string UTF8String]);
[string release];
} // LogIt
just customize the printf to suit your needs
You can use printf(), but the time won't be displayed, and you won't be able to use the "%#" sequence for objects.
That said, you can implement your own logging function, using printf(), and adding support for objects. You will need to know how to deal with C variable arguments.
you can use some trick. use special unicode character like:u00a0
NSLog(#"\n\n\n\u00a0")
Related
Maybe this will be obviously simple for most of you, but could you please give an example how to create similar methods (in Objective-C) and functions in C to create functions like NSString's stringWithFormat:, or NSLog().
Just to remind:
[NSString stringWithFormat:#"example tekst %i %# %.2f", 122, #"sth", 3.1415"];
NSLog(#"account ID %i email %#", accountID, email);
I'd like to create the similar to NSString's method stringWithFormat:, NSURL - urlWithFormat.
What these are called, generally, is "variadic functions" (or methods, as it were).
To create this, simply end your method declartion with , ..., as in
- (void)logMessage:(NSString *)message, ...;
At this point you probably want to wrap it in a printf-like function, as implementing one of those from scratch is trying, at best.
- (void)logMessage:(NSString *)format, ... {
va_list args;
va_start(args, format);
NSLogv(format, args);
va_end(args);
}
Note the use of NSLogv and not NSLog; consider NSLog(NSString *, ...); vs NSLogv(NSString *, va_list);, or if you want a string; initWithFormat:arguments: on NSString *.
If, on the other hand, you are not working with strings, but rather something like
+ (NSArray *)arrayWithObjects:(id)object, ... NS_REQUIRES_NIL_TERMINATION;
things get a lot easier.
In that case, instead of a vprintf-style function, use a loop going through args, assuming id as you go, and parse them as you would in any loop.
- (void)logMessage:(NSString *)format, ... {
va_list args;
va_start(args, format);
id arg = nil;
while ((arg = va_arg(args,id))) {
/// Do your thing with arg here
}
va_end(args);
}
This last sample, of course, assumes that the va_args list is nil-terminated.
Note: In order to make this work you might have to include <stdarg.h>; but if memory serves, this gets included in connection with NSLogv, meaning it comes down by way of "Foundation.h", therefore also "AppKit.h" and "Cocoa.h", as well as a number of others; so this should work out of the box.
- (void)methodWithFormat:(NSString*)format, ... {
va_list args;
va_start(args,format);
//loop, get every next arg by calling va_arg(args,<type>)
// e.g. NSString *arg=va_arg(args,NSString*) or int arg=(args,int)
va_end(args);
}
If you want to pass the variable arguments to stringWithFormat:, use something like:
NSString *s=[[[NSString alloc] initWithFormat:format arguments:args] autorelease];
One thing to mention here is that, the first NSString parameter here comes as format, and the other are passed in the variable argument. right? So before entering the for loop, you have one parameter to handle.
- (NSString *) append:(NSString *)list, ...
{
NSMutableString * res = [NSMutableString string];
[res appendString:list];
va_list args;
va_start(args, list);
id arg = nil;
while(( arg = va_arg(args, id))){
[res appendString:arg];
}
va_end(args);
return res;
}
- (void) test_va_arg
{
NSString * t = [self append:#"a", #"b", #"c", nil];
STAssertEqualObjects(#"abc", t, #"");
}
To ensure that a formatted string returned by NSString initWithFormat:arguments: is as expected, I need to determine if there are the same number of format specifiers as arguments. Below is a (slightly contrived and highly edited) example:
- (void)thingsForStuff:(CustomStuff)stuff, ...
{
NSString *format;
switch (stuff)
{
case CustomStuffTwo:
format = #"Two things: %# and %#";
break;
case CustomStuffThree:
format = #"Three things: %#, %#, and %#";
break;
default:
format = #"Just one thing: %#";
break;
}
va_list args;
va_start(args, method);
// Want to check if format has the same number of %#s as there are args, but not sure how
NSString *formattedStuff = [[NSString alloc] initWithFormat:format arguments:args];
va_end(args);
NSLog(#"Things: %#", formattedStuff);
}
Using this method, [self thingsForStuff:CustomStuffTwo, #"Hello", #"World"] would log
"Two things: Hello and World"
...but [self thingsForStuff:CustomStuffTwo, #"Hello"] would log
"Two things: Hello and "
...something that would be preferred to be caught before it happens.
Is there a way to count the format specifiers in a string, preferably something lightweight/inexpensive?
Well, I created my own regex, I have no idea if it's going to catch all of them, and may end finding some false positives, but seems to be working for me:
static NSString *const kStringFormatSpecifiers =
#"%(?:\\d+\\$)?[+-]?(?:[lh]{0,2})(?:[qLztj])?(?:[ 0]|'.{1})?\\d*(?:\\.\\d+)?[#dDiuUxXoOfeEgGcCsSpaAFn]";
You can count the number of arguments using:
NSRegularExpression *regEx = [NSRegularExpression regularExpressionWithPattern: kStringFormatSpecifiers options:0 error:nil];
NSInteger numSpecifiers = [regEx numberOfMatchesInString: yourString options:0 range:NSMakeRange(0, yourString.length)];
Is there a way to count the format specifiers in a string, preferably
something lightweight/inexpensive?
Nope -- really isn't. At least, not if you want it to work across all possible format strings. You would have to duplicate the parser that is used by stringWithFormat:. I.e. don't try to validate everything.
You could count the number of %, but that would not catch things like %% or other special cases. That may be good enough for your purposes.
Because of the way C and Objective-C handle variadic functions/methods like yours, you cannot in general tell how many arguments the user has provided.
Here are two ways to handle your situation.
First, look for another way to do this. The number of arguments you pass to the method is determined at compile-time. So maybe instead of using a variadic method, you should just have three methods:
- (void)doStuff:(CustomStuff)stuff withThing:(Thing *)thing;
- (void)doStuff:(CustomStuff)stuff withThing:(Thing *)thing1 thing:(Thing *)thing2;
- (void)doStuff:(CustomStuff)stuff withThing:(Thing *)thing1 thing:(Thing *)thing2 hatWearer:(Cat *)cat;
And you select the right method to call at compile-time based on how many arguments you want to pass, eliminating the switch statement entirely.
Second, I see that your predefined format strings only use the %# format. Does this mean that you expect the user to only pass objects to your method (aside from the (CustomStuff)stuff argument)?
If the user will only pass objects to your method, and you require those arguments to be non-nil, then you can get the compiler to help you out. Change your method to require the user to pass nil at the end of the argument list. You can tell the compiler that the argument list has to be nil-terminated by declaring the method (in your #interface) like this:
#interface MyObject : NSObject
- (void)thingsForStuff:(CustomStuff)stuff, ... NS_REQUIRES_NIL_TERMINATION
#end
Now the compiler will warn the user “Missing sentinel in method dispatch” if he calls your method without putting a literal nil at the end of the argument list.
So, having changed your API to require some non-nil arguments followed by a nil argument, you can change your method to count up the non-nil arguments like this:
- (void)thingsForStuff:(CustomStuff)stuff, ... {
int argCount = 0;
va_list args;
va_start(args, stuff);
while (va_arg(args, id)) {
++argCount;
}
va_end(args)
int expectedArgCount;
NSString *format;
switch (stuff) {
case CustomStuffTwo:
expectedArgCount = 2;
format = #"Two things: %# and %#";
break;
case CustomStuffThree:
expectedArgCount = 3;
format = #"Three things: %#, %#, and %#";
break;
// etc.
}
NSAssert(argCount == expectedArgCount, #"%# %s called with %d non-nil arguments, but I expected %d", self, (char*)_cmd, argCount, expectedArgCount);
va_start(args, stuff);
NSString *formattedStuff = [[NSString alloc] initWithFormat:format arguments:args];
va_end(args);
NSLog(#"Things: %#", formattedString);
}
You could count the number of format specifiers, but IIRC you will never be able to count the number of arguments passed into a variable-argument method. This is because of the way C pushes arguments on the stack without specifying how many it has pushed.
Most functions overcome this by requiring that the last argument be nil or some kind of terminator (see [NSArray arrayWithObjects:]). There's even a macro that allows the compiler to check this and emit a warning at compile time.
You can use NS_FORMAT_FUNCTION at the end of your function prototype, as in stringWithFormat method of NSString.
So your method's prototype should be like this:
- (void)thingsForStuff:(CustomStuff)stuff, ... NS_FORMAT_FUNCTION(1,2);
long specifierCount = [myFormatString componentsSeparatedByString:#"%"].count;
This will get you close. Its just a simple split. You would have to account for escaped % values.
I am trying to use a custom logger to give clean console output for a text-based game. Below is the code of the method in question:
void MyLog(NSString *format, ...)
{
va_list args;
va_start(args, format);
NSString *formattedString = [[NSString alloc] initWithFormat:format arguments: args];
va_end(args);
[[NSFileHandle fileHandleWithStandardOutput] writeData:[formattedString dataUsingEncoding: NSNEXTSTEPStringEncoding]];
[formattedString release];
}
and here is this example's use of it:
NSLog(#"Unicode text: \u2665");
NSLog(#"ASCII Text");
MyLog(#"Unicode text: \u2665");
MyLog(#"ASCII text");
and the output:
2012-04-26 00:08:53.614 TextCraft[11319:a0f] Unicode text: ♥
2012-04-26 00:08:53.617 TextCraft[11319:a0f] ASCII Text
ASCII text
As you can see, the MyLog function works fine for low-page characters, but when it tries to print high-page characters, it simply skips the whole line. Does anyone know why this is?
You're using NSNEXTSTEPStringEncoding in your call to -dataUsingEncoding:, which is just ASCII plus some additions. If you want to support Unicode, you should use an appropriate encoding like NSUTF8StringEncoding.
I am trying to duplicate NSLog but without all of the unnecessary dates at the beginning. I have tried the following c function (I made myself), but it wont log any other values that are not NSStrings. Please could you tell me how I could do it so that it would log any value?
static void echo(NSString *fmt, ...) {
printf("<<<<<<<%s>>>>>>>", [fmt UTF8String]);
}
To use variadic argument lists in C you need to use a few macros that are defined in the stdarg.h header file that comes with your compiler.
here is a detailed explanation of how to write your own printf
If you just want to pass the arguments to the real printf without further manipulation you can use the vfprintf variant of printf instead but you need to extend the fmt parameter separately:
static void echo(NSString *fmt, ...)
{
va_list args;
NSString *logfmt = [NSString stringWithFormat: #"<<<<<<<%s>>>>>>>", [fmt UTF8String]];
va_start (args, fmt);
vfprintf( stdout, [logfmt UTF8String], args );
va_end (args);
[logfmt release];
}
What is the best way to tokenize/split a NSString in Objective-C?
Found answer here:
NSString *string = #"oop:ack:bork:greeble:ponies";
NSArray *chunks = [string componentsSeparatedByString: #":"];
Everyone has mentioned componentsSeparatedByString: but you can also use CFStringTokenizer (remember that an NSString and CFString are interchangeable) which will tokenize natural languages too (like Chinese/Japanese which don't split words on spaces).
If you just want to split a string, use -[NSString componentsSeparatedByString:]. For more complex tokenization, use the NSScanner class.
If your tokenization needs are more complex, check out my open source Cocoa String tokenizing/parsing toolkit: ParseKit:
http://parsekit.com
For simple splitting of strings using a delimiter char (like ':'), ParseKit would definitely be overkill. But again, for complex tokenization needs, ParseKit is extremely powerful/flexible.
Also see the ParseKit Tokenization documentation.
If you want to tokenize on multiple characters, you can use NSString's componentsSeparatedByCharactersInSet. NSCharacterSet has some handy pre-made sets like the whitespaceCharacterSet and the illegalCharacterSet. And it has initializers for Unicode ranges.
You can also combine character sets and use them to tokenize, like this:
// Tokenize sSourceEntityName on both whitespace and punctuation.
NSMutableCharacterSet *mcharsetWhitePunc = [[NSCharacterSet whitespaceAndNewlineCharacterSet] mutableCopy];
[mcharsetWhitePunc formUnionWithCharacterSet:[NSCharacterSet punctuationCharacterSet]];
NSArray *sarrTokenizedName = [self.sSourceEntityName componentsSeparatedByCharactersInSet:mcharsetWhitePunc];
[mcharsetWhitePunc release];
Be aware that componentsSeparatedByCharactersInSet will produce blank strings if it encounters more than one member of the charSet in a row, so you might want to test for lengths less than 1.
If you're looking to tokenise a string into search terms while preserving "quoted phrases", here's an NSString category that respects various types of quote pairs: "" '' ‘’ “”
Usage:
NSArray *terms = [#"This is my \"search phrase\" I want to split" searchTerms];
// results in: ["This", "is", "my", "search phrase", "I", "want", "to", "split"]
Code:
#interface NSString (Search)
- (NSArray *)searchTerms;
#end
#implementation NSString (Search)
- (NSArray *)searchTerms {
// Strip whitespace and setup scanner
NSCharacterSet *whitespace = [NSCharacterSet whitespaceAndNewlineCharacterSet];
NSString *searchString = [self stringByTrimmingCharactersInSet:whitespace];
NSScanner *scanner = [NSScanner scannerWithString:searchString];
[scanner setCharactersToBeSkipped:nil]; // we'll handle whitespace ourselves
// A few types of quote pairs to check
NSDictionary *quotePairs = #{#"\"": #"\"",
#"'": #"'",
#"\u2018": #"\u2019",
#"\u201C": #"\u201D"};
// Scan
NSMutableArray *results = [[NSMutableArray alloc] init];
NSString *substring = nil;
while (scanner.scanLocation < searchString.length) {
// Check for quote at beginning of string
unichar unicharacter = [self characterAtIndex:scanner.scanLocation];
NSString *startQuote = [NSString stringWithFormat:#"%C", unicharacter];
NSString *endQuote = [quotePairs objectForKey:startQuote];
if (endQuote != nil) { // if it's a valid start quote we'll have an end quote
// Scan quoted phrase into substring (skipping start & end quotes)
[scanner scanString:startQuote intoString:nil];
[scanner scanUpToString:endQuote intoString:&substring];
[scanner scanString:endQuote intoString:nil];
} else {
// Single word that is non-quoted
[scanner scanUpToCharactersFromSet:whitespace intoString:&substring];
}
// Process and add the substring to results
if (substring) {
substring = [substring stringByTrimmingCharactersInSet:whitespace];
if (substring.length) [results addObject:substring];
}
// Skip to next word
[scanner scanCharactersFromSet:whitespace intoString:nil];
}
// Return non-mutable array
return results.copy;
}
#end
If you are looking for splitting linguistic feature's of a string (Words, paragraphs, characters, sentences and lines), use string enumeration:
NSString * string = #" \n word1! word2,%$?'/word3.word4 ";
[string enumerateSubstringsInRange:NSMakeRange(0, string.length)
options:NSStringEnumerationByWords
usingBlock:
^(NSString *substring, NSRange substringRange, NSRange enclosingRange, BOOL *stop) {
NSLog(#"Substring: '%#'", substring);
}];
// Logs:
// Substring: 'word1'
// Substring: 'word2'
// Substring: 'word3'
// Substring: 'word4'
This api works with other languages where spaces are not always the delimiter (e.g. Japanese). Also using NSStringEnumerationByComposedCharacterSequences is the proper way to enumerate over characters, since many non-western characters are more than one byte long.
I had a case where I had to split the console output after an LDAP query with ldapsearch. First set up and execute the NSTask (I found a good code sample here: Execute a terminal command from a Cocoa app). But then I had to split and parse the output so as to extract only the print-server names out of the Ldap-query-output. Unfortunately it is rather tedious string-manipulation which would be no problem at all if we were to manipulate C-strings/arrays with simple C-array operations. So here is my code using cocoa objects. If you have better suggestions, let me know.
//as the ldap query has to be done when the user selects one of our Active Directory Domains
//(an according comboBox should be populated with print-server names we discover from AD)
//my code is placed in the onSelectDomain event code
//the following variables are declared in the interface .h file as globals
#protected NSArray* aDomains;//domain combo list array
#protected NSMutableArray* aPrinters;//printer combo list array
#protected NSMutableArray* aPrintServers;//print server combo list array
#protected NSString* sLdapQueryCommand;//for LDAP Queries
#protected NSArray* aLdapQueryArgs;
#protected NSTask* tskLdapTask;
#protected NSPipe* pipeLdapTask;
#protected NSFileHandle* fhLdapTask;
#protected NSMutableData* mdLdapTask;
IBOutlet NSComboBox* comboDomain;
IBOutlet NSComboBox* comboPrinter;
IBOutlet NSComboBox* comboPrintServer;
//end of interface globals
//after collecting the print-server names they are displayed in an according drop-down comboBox
//as soon as the user selects one of the print-servers, we should start a new query to find all the
//print-queues on that server and display them in the comboPrinter drop-down list
//to find the shares/print queues of a windows print-server you need samba and the net -S command like this:
// net -S yourPrintServerName.yourBaseDomain.com -U yourLdapUser%yourLdapUserPassWord -W adm rpc share -l
//which dispalays a long list of the shares
- (IBAction)onSelectDomain:(id)sender
{
static int indexOfLastItem = 0; //unfortunately we need to compare this because we are called also if the selection did not change!
if ([comboDomain indexOfSelectedItem] != indexOfLastItem && ([comboDomain indexOfSelectedItem] != 0))
{
indexOfLastItem = [comboDomain indexOfSelectedItem]; //retain this index for next call
//the print-servers-list has to be loaded on a per univeristy or domain basis from a file dynamically or from AN LDAP-QUERY
//initialize an LDAP-Query-Task or console-command like this one with console output
/*
ldapsearch -LLL -s sub -D "cn=yourLdapUser,ou=yourOuWithLdapUserAccount,dc=yourDomain,dc=com" -h "yourLdapServer.com" -p 3268 -w "yourLdapUserPassWord" -b "dc=yourBaseDomainToSearchIn,dc=com" "(&(objectcategory=computer)(cn=ps*))" "dn"
//our print-server names start with ps* and we want the dn as result, wich comes like this:
dn: CN=PSyourPrintServerName,CN=Computers,DC=yourBaseDomainToSearchIn,DC=com
*/
sLdapQueryCommand = [[NSString alloc] initWithString: #"/usr/bin/ldapsearch"];
if ([[comboDomain stringValue] compare: #"firstDomain"] == NSOrderedSame) {
aLdapQueryArgs = [NSArray arrayWithObjects: #"-LLL",#"-s", #"sub",#"-D", #"cn=yourLdapUser,ou=yourOuWithLdapUserAccount,dc=yourDomain,dc=com",#"-h", #"yourLdapServer.com",#"-p",#"3268",#"-w",#"yourLdapUserPassWord",#"-b",#"dc=yourFirstDomainToSearchIn,dc=com",#"(&(objectcategory=computer)(cn=ps*))",#"dn",nil];
}
else {
aLdapQueryArgs = [NSArray arrayWithObjects: #"-LLL",#"-s", #"sub",#"-D", #"cn=yourLdapUser,ou=yourOuWithLdapUserAccount,dc=yourDomain,dc=com",#"-h", #"yourLdapServer.com",#"-p",#"3268",#"-w",#"yourLdapUserPassWord",#"-b",#"dc=yourSecondDomainToSearchIn,dc=com",#"(&(objectcategory=computer)(cn=ps*))",#"dn",nil];
}
//prepare and execute ldap-query task
tskLdapTask = [[NSTask alloc] init];
pipeLdapTask = [[NSPipe alloc] init];//instead of [NSPipe pipe]
[tskLdapTask setStandardOutput: pipeLdapTask];//hope to get the tasks output in this file/pipe
//The magic line that keeps your log where it belongs, has to do with NSLog (see https://stackoverflow.com/questions/412562/execute-a-terminal-command-from-a-cocoa-app and here http://www.cocoadev.com/index.pl?NSTask )
[tskLdapTask setStandardInput:[NSPipe pipe]];
//fhLdapTask = [[NSFileHandle alloc] init];//would be redundand here, next line seems to do the trick also
fhLdapTask = [pipeLdapTask fileHandleForReading];
mdLdapTask = [NSMutableData dataWithCapacity:512];//prepare capturing the pipe buffer which is flushed on read and can overflow, start with 512 Bytes but it is mutable, so grows dynamically later
[tskLdapTask setLaunchPath: sLdapQueryCommand];
[tskLdapTask setArguments: aLdapQueryArgs];
#ifdef bDoDebug
NSLog (#"sLdapQueryCommand: %#\n", sLdapQueryCommand);
NSLog (#"aLdapQueryArgs: %#\n", aLdapQueryArgs );
NSLog (#"tskLdapTask: %#\n", [tskLdapTask arguments]);
#endif
[tskLdapTask launch];
while ([tskLdapTask isRunning]) {
[mdLdapTask appendData: [fhLdapTask readDataToEndOfFile]];
}
[tskLdapTask waitUntilExit];//might be redundant here.
[mdLdapTask appendData: [fhLdapTask readDataToEndOfFile]];//add another read for safety after process/command stops
NSString* sLdapOutput = [[NSString alloc] initWithData: mdLdapTask encoding: NSUTF8StringEncoding];//convert output to something readable, as NSData and NSMutableData are mere byte buffers
#ifdef bDoDebug
NSLog(#"LdapQueryOutput: %#\n", sLdapOutput);
#endif
//Ok now we have the printservers from Active Directory, lets parse the output and show the list to the user in its combo box
//output is formatted as this, one printserver per line
//dn: CN=PSyourPrintServer,OU=Computers,DC=yourBaseDomainToSearchIn,DC=com
//so we have to search for "dn: CN=" to retrieve each printserver's name
//unfortunately splitting this up will give us a first line containing only "" empty string, which we can replace with the word "choose"
//appearing as first entry in the comboBox
aPrintServers = (NSMutableArray*)[sLdapOutput componentsSeparatedByString:#"dn: CN="];//split output into single lines and store it in the NSMutableArray aPrintServers
#ifdef bDoDebug
NSLog(#"aPrintServers: %#\n", aPrintServers);
#endif
if ([[aPrintServers objectAtIndex: 0 ] compare: #"" options: NSLiteralSearch] == NSOrderedSame){
[aPrintServers replaceObjectAtIndex: 0 withObject: slChoose];//replace with localized string "choose"
#ifdef bDoDebug
NSLog(#"aPrintServers: %#\n", aPrintServers);
#endif
}
//Now comes the tedious part to extract only the print-server-names from the single lines
NSRange r;
NSString* sTemp;
for (int i = 1; i < [aPrintServers count]; i++) {//skip first line with "choose". To get rid of the rest of the line, we must isolate/preserve the print server's name to the delimiting comma and remove all the remaining characters
sTemp = [aPrintServers objectAtIndex: i];
sTemp = [sTemp stringByTrimmingCharactersInSet: [NSCharacterSet whitespaceAndNewlineCharacterSet]];//remove newlines and line feeds
#ifdef bDoDebug
NSLog(#"sTemp: %#\n", sTemp);
#endif
r = [sTemp rangeOfString: #","];//now find first comma to remove the whole rest of the line
//r.length = [sTemp lengthOfBytesUsingEncoding:NSUTF8StringEncoding];
r.length = [sTemp length] - r.location;//calculate number of chars between first comma found and lenght of string
#ifdef bDoDebug
NSLog(#"range: %i, %i\n", r.location, r.length);
#endif
sTemp = [sTemp stringByReplacingCharactersInRange:r withString: #"" ];//remove rest of line
#ifdef bDoDebug
NSLog(#"sTemp after replace: %#\n", sTemp);
#endif
[aPrintServers replaceObjectAtIndex: i withObject: sTemp];//put back string into array for display in comboBox
#ifdef bDoDebug
NSLog(#"aPrintServer: %#\n", [aPrintServers objectAtIndex: i]);
#endif
}
[comboPrintServer removeAllItems];//reset combo box
[comboPrintServer addItemsWithObjectValues:aPrintServers];
[comboPrintServer setNumberOfVisibleItems:aPrintServers.count];
[comboPrintServer selectItemAtIndex:0];
#ifdef bDoDebug
NSLog(#"comboPrintServer reloaded with new values.");
#endif
//release memory we used for LdapTask
[sLdapQueryCommand release];
[aLdapQueryArgs release];
[sLdapOutput release];
[fhLdapTask release];
[pipeLdapTask release];
// [tskLdapTask release];//strangely can not be explicitely released, might be autorelease anyway
// [mdLdapTask release];//strangely can not be explicitely released, might be autorelease anyway
[sTemp release];
}
}
I have my self come across instance where it was not enough to just separate string by component many tasks such as 1) Categorizing token into types 2) Adding new tokens 3)Separating string between custom closures like all words between "{" and "}"For any such requirements i found Parse Kit a life saver.
I used it to parse .PGN (prtable gaming notation) files successfully its very fast and lite.