Objective-C - FMDB - Large SQLite Dump Import - objective-c

I have a large SQLite file filled up with queries to create my database tables and insert all records. The file is rather large and running the SQL file seems to take much longer than I would have expected.
I am using FMDB for an iPad app I am working on, and I really want to just replace the current DB file with a new one, but I am not sure that an sql file is the same as a DB file. It doens't contain any of the same header info, etc...
What's the best way to go about doing this?

If doing a lot of separate UPDATE or INSERT calls with FMDatabase, consider doing beginTransaction at the start and commit at the end:
[db beginTransaction];
// do all of your updates
[db commit];
Or, if using FMDatabaseQueue, use inTransaction:
[databaseQueue inTransaction:^(FMDatabase *db , BOOL *rollback) {
// do all of your updates
}];
If you don't use one of those, it will commit after each insert, which makes it much slower. The difference can be dramatic if adding lots of rows (e.g. I've seen performance differences of two orders of magnitude when adding/updating lots of little records).
The above assumes that you are trying to perform a series of separate SQL commands. If it's all in one file (such as .dump output), FMDB hasn't historically had an interface to do that (even though there is a SQLite function, sqlite3_exec, that does precisely this). There was a recent addition to the extra folder called FMDatabaseSplitter, which attempts to splits a long string of SQL into separate calls which you can then invoke separately.
Personally, it makes me nervous to use a third-party SQL parsing routine, so I'd just be inclined to call the SQLite function sqlite3_exec directly. To do that, you can access the sqlite3 pointer from your FMDatabase object using the FMDB sqliteHandle method, and then use that in conjunction with the sqlite3_exec function directly:
NSError *error = nil;
NSString *dumpSQL = [NSString stringWithContentsOfFile:dumpFilePath encoding:NSUTF8StringEncoding error:&error];
NSAssert(dumpSQL, #"Loading of SQL failed: %#", error);
int rc = sqlite3_exec(db.sqliteHandle, [dumpSQL UTF8String], NULL, NULL, NULL);
if (rc != SQLITE_OK) {
NSLog(#"sqlite3_exec error: %#", [db lastErrorMessage]);
}
I must confess, it makes me a little nervous to just take bulk SQL an import it into an app's database. An innocent mistake in the SQL could brick the app of your entire install-base if you're not extremely careful. I'd rather see the app request JSON or XML feed from the server, and then do the updates itself, but if you want to use the .dump output to update the app's database with FMDB, this is one way to do it.
FMDB v2.3 has introduced a wrapper for sqlite3_exec called executeStatements:
BOOL success;
NSString *sql = #"create table bulktest1 (id integer primary key autoincrement, x text);"
"create table bulktest2 (id integer primary key autoincrement, y text);"
"create table bulktest3 (id integer primary key autoincrement, z text);"
"insert into bulktest1 (x) values ('XXX');"
"insert into bulktest2 (y) values ('YYY');"
"insert into bulktest3 (z) values ('ZZZ');";
success = [db executeStatements:sql];

Related

Is it possible to search SQLite database that has base64-encoded some text strings?

I know you can query a dataBase like so
NSString *strSelectQuery = [NSString stringWithFormat: #"SELECT * FROM Songs WHERE songname LIKE '%%%#%%'", str];
However, the problem is I encoded the strings (base64) before storing the data into my sqlite database. So regular searches from search bar won't work in this situation. I have another solution that grabs all the Data and stores it in an array then I can predicate it. However, I am wondering if there is a way to search data using the query above somehow even when the strings are encoded by base64.
In chat with you, it became clear that the previous developer was having trouble inserting values into the database, so he chose to base64-encode those strings.
Bottom line, searching for substring within base64 string is technically possible, but surprisingly complicated and highly inefficient. I would not advise attempting to doing so.
Instead I would not base64-encode the strings in the database at all. The database will be smaller (base64-encoded string is 33% larger than original payload), the code will be simpler, and searching will now work without any effort at all.
You mentioned that you encountered errors when trying to insert text values into your database. This is because using stringWithFormat to build SQL can cause problems. Consider:
NSString *sql = [NSString stringWithFormat: #"INSERT INTO book (title) VALUES ('%#')", title];
BOOL success = [db executeUpdate:sql]; // FMDB method
But that would fail if the title contained an apostrophe:
Finnegan's Wake
Yes, base64-encoding could get around that problem, but, better, you could just replace that with SQL with ? placeholders, too:
NSString *sql = #"INSERT INTO book (title) VALUES (?)";
BOOL success = [db executeUpdate:sql, title];
By the way, to search, you'd also want to use ? placeholders:
NSString *sql = #"SELECT * FROM book WHERE title LIKE ?";
NSString *searchTerm = [NSString stringWithFormat:#"%%%#%%", searchString];
FMResultSet *rs = [db executeQuery:sql, searchTerm];

Suggestions to Process Complex + Large Data File

I have a very large and complex data file (.txt, see snippet below) of about 10MB and would like to know the best way to store it and access it later on.
My app currently uses core data for storage of other entities but I don't see how I can create an entity from this type of data file because of its complexity.
This file is divided as follows:
First line of each major section begins with an A| and means a new 'airway' to be defined. Then, is it's name, so in the example below we have the airway named V320 and another named V321. On the following lines, we have important data, the 'points'/waypoints which make up this airway. Each one has a name, and coordinates. So the first one here is PLN at 45.63N and -84.66W (coordinates). Then, from there the next one is LORIW at 45.35N and -84.92W, from LORIW we go to IROTO, and so on...
NOTE: There may be two, three, maybe even 4 airways with the same 'name' like V320 for example has 3...but each one is in it's own part of the map.
The other values there are irrelevant such as the numbers after the coordinate pair.
In essence, I need all this so that I can then draw lines on my map (GMSPolyLine using Google map SDK) which goes through all these points for each airway and then to create GMSMarkers(google version of MKAnnotation) for each waypoint which the user can tap.
I can handle the drawing of lines/markers on the map but the difficult part for me to visualize is the manipulation of this data and making it easier to access.
Let me know if you have any questions.
A|V320|20
S|PLN|045630647|-0084664108|LORIW|045352072|-0084924214|0|219|1998
S|LORIW|045352072|-0084924214|IROTO|045188989|-0085075111|219|219|1168
S|IROTO|045188989|-0085075111|ADENO|045030644|-0085220425|219|219|1132
S|ADENO|045030644|-0085220425|TIDDU|044877978|-0085359767|215|215|1090
S|TIDDU|044877978|-0085359767|SKIPR|044831714|-0085401772|215|215|330
.....
A|V321|29
S|PZD|031655206|-0084293100|KUTVE|031866950|-0084451303|0|329|1505
S|KUTVE|031866950|-0084451303|DUVAT|031948772|-0084512695|329|329|582
S|DUVAT|031948772|-0084512695|LUMPP|032041158|-0084582139|329|329|657
S|LUMPP|032041158|-0084582139|PREST|032176375|-0084684117|329|329|963
S|PREST|032176375|-0084684117|CSG|032615253|-0085017631|326|326|3129
S|CSG|032615253|-0085017631|JALVO|032722436|-0085064033|326|339|684
.....
Your data exhibits some regularity. If it is predictable and consistent, just write a parser that iterates through the file and creates appropriate Core Data entities.
For example, the fact that each new airway is separated by a newline can help you find those. Also, each final waypoint is repeated in the next line unless you are at the end of an airway record. I think you can do this in maybe 20-30 lines of code.
On your development machine (or even on an iPad or recent iPhone, for that matter), even creating a 10MB array in memory (to be parsed) should not be a constraint.
If the data is static, you can use the resulting sqlite database as a read-only persistent store that you can include in your app bundle.
As for the parser, it would be something like this:
NSString *file = [[NSString alloc] initWithContentsOfFile:fileURLString
encoding:NSUTF8StringEncoding error:nil];
NSArray *lines = [file componentsSeparatedByString:#"\n"];
for (NSString *line in lines) {
if (line.length < 1) { continue; }
NSArray *fields = [line componentsSeparatedByString:#"|"];
if ([fields.firstObject isEqualToString:#"A"]) {
// insert new airway object and populate with other fields
}
else if ([fields.firstObject isEqualToString:#"S"]) {
// insert new waypoint object (two for each first line)
// assign as relationship to the current airway
// and to another waypoint as necessary
}
}
[managedObjectContext save:nil];

Sqlite3 database in iPhone gets locked - how to avoid?

I have a query that performs a search on an Sqlite3 DB. It does nothing but read using a reader.
For each found match it calls a callback to the UI which updates a result view.
While this search is running, I hit a button in the UI which will perform some other action in a new thread. In the end it is supposed to remove the search controller's view and show a new controller.
However, at some point the the triggered action wants to write to the databse. And there it just hangs and eventually I will see an exception that the DB is locked.
Interesting is also, that the search reader does not continue either, it is a deadlock.
Do I have to open the database in some special way to support multithreaded usage? What would the constructor for the connection be?
MonoTouch 5.1+ provides an API to let you select the threading model to be used with SQLite.
SqliteConnection.SetConfig (SQLiteConfig.MultiThread);
This maps to some of the connection options of SQLite library.
UPDATE: If you're using an earlier version of MonoTouch (e.g. between 4.2 and 5.0.x) you can use the binary attached to the bug report #652 (follow the instructions) or copy-paste the patch (p/invoke and enum) inside your own application.
I'm not sure I interpret your description correctly, but the way you describe it, it sounds to me that your "reader" steps through the database row-by-row and every time it finds a result it does a callback to a callback function? Is this correct?
If that is the case, you might repeatedly lock your DB, and your search will be slow.
The right way is to extract all matches into a result set in one single query - once that query is completed the lock will be released and you have a result set from SQL that contains only the matching rows.
You let SQLite create a result set like this by using a query of the type "SELECT * FROM tablename WHERE columnX LIKE '%searchstring%'"
(or similar, depending on your search criteria)
This will create a result set with all matches in the database and then release the database lock. Then you can step through the result and create objects and put into and NSArray that is connected to your UI view.
NSArray retval = [NSMutableArray array];
//Create a query
NSString *query = [NSString stringWithFormat:#"SELECT * FROM %# WHERE %# LIKE %#",
tableName, columnName, searchString];
sqlite3_stmt *statement;
//Database locks here
if (sqlite3_prepare_v2(_database, [query UTF8String], -1, &statement, nil)
== SQLITE_OK) {
//Database should unlock here, the query is finished
while (sqlite3_step(statement) == SQLITE_ROW) {
char *nameChars = (char *) sqlite3_column_text(statement, 0);
NSString *name = [NSString stringWithUTF8String:nameChars];
SomeClass *info = [[SomeClass alloc] initWithName:name];
/* Extract other columns and put in your object */
[retval addObject:info];
[info release];
}
sqlite3_finalize(statement);
} else {
NSLog(#"SQL-statement failed");
}
Doing this way there shouldn't be a problem to write to the DB when it is necessary. Only perform new queries to the DB when it's absolutely necessary, for example when your search criteria changed or the content in the DB has been updated.
Do not run repeated queries to a DB that has not changed, or with unchanged search criteria.

How to serialize NSImage to an sql script in order to import it in a blob column?

My iphone app uses a SQLite database to store data. I have a table with a blob column where i store my images.
When i do an update i don't want to overwrite the user's database, i want to execute some sql scripts and inject new data if needed.
I have a utility app made for Mac that should make the sql scripts that will be run on the iphone.
I have the images stored as NSImages in my app but i have problems when i want to export the data as sql scripts(simple text files).
My files should have lines like:
Insert into Images(imageData) values ( ___IMAGE1_DATA___ );
Insert into Images(imageData) values ( ___IMAGE2_DATA___ );
Insert into Images(imageData) values ( ___IMAGE3_DATA___ );
the question is how could i serialize images data to my sql script in order to import the data correctly into the blob column?
You can use TIFFRepresentation of the NSImage to get hold of the NSData representation.
Actually I would save the images to disk and reference those only from your sqlite database. This can improve performance when your images tend to be large.
You will need to get the underlying CGImage object, using the CGImage method of UIImage. Then take a look at the CGImage reference:
http://developer.apple.com/library/mac/#documentation/GraphicsImaging/Reference/CGImage/Reference/reference.html
Then use a CGContextRef object to draw the image, and obtain the pixel data.
There's a good example on StackOverflow:
How to get pixel data from a UIImage (Cocoa Touch) or CGImage (Core Graphics)?
You can also use the UIImagePNGRepresentationor UIImagePNGRepresentation functions, that return directly a NSData object, if those formats suits you. It will be a lot simpler.
Then use the bytesmethod of NSData to have a pointer to the image data (const void *).
I found a solution to serialize the images to an sql script and then insert them into the db (blob column).
I extract the NSData from an image like this NSData *thumbailData = [thumbnail TIFFRepresentation]; (thanks Nick).
After i extract the NSData, i convert it into a hex string using the method below. I added it to a category of NSData.
- (NSString*) hexString {
NSMutableString *stringBuffer = [NSMutableString
stringWithCapacity:([self length] * 2)];
const unsigned char *dataBuffer = [self bytes];
int i;
for (i = 0; i < [self length]; ++i)
[stringBuffer appendFormat:#"%02x", (unsigned long)dataBuffer[ i ]];
return [[stringBuffer copy] autorelease];
}
NSString *hexRepresentation = [thumbnailData hexString];
The hexRepresentation will look like below:
4d4d002a00005a48fafafafff8f8f8fff8f8f8fff9f9f9fff8f8f8fff8f8f8
…
In order to serialize the hexRepresentation of the image i created an SQL script like below:
INSERT INTO Thumbnails (Picture_uid, Thumbnail) Values(10, x'4d4d002a00005a48fafafafff8f8f8fff8f8f8fff9f9f9fff8f8f8fff8f8f8 … ‘) ;
the x' data ' tells the db that it will receive info in hex format and it will know how to deal with it.
one of the problems with this solution is that it will double the size of the script. if you'll have an image of 200kb the script will have 400kb but in the db the image will be 200kb.
for me this was a good solution to update my db using sql scripts without writing any code.

Reading and writing images to an SQLite DB for iPhone use

I've set up a SQLite DB that currently reads and writes NSStrings perfectly. I also want to store an image in the database and recall it later. I've read up a bit on using NSData and encoding the image, but I'm not entirely sure what the syntax is for what I want to do. Any code snippets or examples would be greatly appreciated.
My current process goes like this:
UIImagePickerController -> User Chooses Image from Photos -> chosenImage is set to instance of UIImageView -> Now I want to take this image and store it in the DB
I should mention this call will eventually be replaced with a call to a remote server. Not sure if this makes a difference as far as performance goes.
You'll need to convert the UIImage hosted within your UIImageView into a binary BLOB for storage in SQLite. To do that, you can use the following:
NSData *dataForImage = UIImagePNGRepresentation(cachedImage);
sqlite3_bind_blob(yourSavingSQLStatement, 2, [dataForImage bytes], [dataForImage length], SQLITE_TRANSIENT);
This will generate a PNG representation of your image, store it in an NSData instance, and then bind the bytes from the NSData as a BLOB for the second argument in your SQL query. Use UIImageJPEGRepresentation in the above to store in that format, if you like. You will need to have a BLOB column added to the appropriate table in your SQLite database.
To retrieve this image, you can use the following:
NSData *dataForCachedImage = [[NSData alloc] initWithBytes:sqlite3_column_blob(yourLoadingSQLStatement, 2) length: sqlite3_column_bytes(yourLoadingSQLStatement, 2)];
self.cachedImage = [UIImage imageWithData:dataForCachedImage];
[dataForCachedImage release];
One option (and generally preferred when working in SQL) is to write the image to a file on the system and store the path (or some other kind of identifier) in the database.
Apple's recommendation is not to store BLOB's in SQLite databases that are bigger than ~2 kilobytes.
SQLite organizes databases into pages. Each page is 4 kilobytes in size. When you read data from the SQLite database file it loads these pages into an internal page cache. On the iPhone I think this cache defaults to 1 megabyte in size. This makes reading adjacent records very fast because they will probably be in the page cache already.
When SQLite reads your database record into memory it reads the entire record and all of the pages that it occupies. So if your record contains a BLOB, it could occupy many pages and you will be ejecting existing pages from the cache and replacing them with your BLOB record's pages.
This isn't so bad if you're just scanning through and loading all your BLOBS to do something with them (display them for example). But if say you did a query where you just wanted to get some data that is in the same row as the BLOB this query would be much slower than if the record did not contain the large BLOB.
So at a minimum you should store your BLOB data in a separate table. Eg:
CREATE TABLE blobs ( id INTEGER PRIMARY KEY, data BLOB );
CREATE TABLE photos ( id INTEGER PRIMARY KEY, name TEXT, blob_id INTEGER,
FOREIGN KEY(blob_id) REFERENCES blobs(id) );
Or better yet, store the BLOB data as files outside of the SQLite database.
Note that it may be possible to tweak the page cache size with SQL PRAGMA statements (if you're not using CoreData).
Writing Image to SQLite DB
if(myImage != nil){
NSData *imgData = UIImagePNGRepresentation(myImage);
sqlite3_bind_blob(update_stmtement, 6, [imgData bytes], [imgData length], NULL);
}
else {
sqlite3_bind_blob(update_stmtement, 6, nil, -1, NULL);
}
Reading From SQLite DB:
NSData *data = [[NSData alloc] initWithBytes:sqlite3_column_blob(init_statement, 6) length:sqlite3_column_bytes(init_statement, 6)];
if(data == nil)
NSLog(#"No image found.");
else
self.pictureImage = [UIImage imageWithData:data];
you should first write image on file system. and then take the image path and store that image path(URL) as TEXT in sqlite.
The best practice is to compress the image and store it in the database or file system. If you don't care about the resolution of the image you can go ahead and even resize the image by using :
UIGraphicsBeginImageContext(newSize);
[image drawInRect:CGRectMake(0,0,newSize.width,newSize.height)];
UIImage* newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
After that you can use UIImageJPEGRepresentation for jpeg images with a value of "0" for maximum compression. Or you can use UIImagePNGRepresentation for png images