Bizarre results with bif:contains - corrupted full text index? - sparql

Question

I recently built a Virtuoso database (version 07.10.3207) using dbpedia data. I'm trying to build some queries for it, and encountering very strange results. For one example:
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
select ?s, ?p, ?o where {
?s ?p ?o .
?s rdfs:label "Almond"#en .
?o bif:contains "mythical"
}
This yields a hit. One might expect it to mean that the comment field (the field that matches "mythical") for Almond contains the word "mythical". However, it does not. It is, in fact:
"The almond (/??m?nd/) (Prunus dulcis, syn. Prunus amygdalus, Amygdalus communis, Amygdalus dulcis) (or badam in Indian English, from Persian: ??????) is a species of tree native to the Middle East and South Asia. "Almond" is also the name of the edible and widely cultivated seed of this tree."#en
Many other queries yield similarly strange results.
Trying the same queries on the public dbpedia endpoint does not yield these bizarre results, so I know it's somehow a problem with my database. I guessed that it might have to do with some corruption of the full text indices.
I tried the following, without a super-clear understanding of what exactly they might do, based on other notes I was able to find:
DB.DBA.RDF_OBJ_FT_RULE_ADD(null, null, 'All');
DB.DBA.VT_INC_INDEX_DB_DBA_RDF_OBJ();
DB.DBA.RDF_OBJ_FT_RECOVER();
DB.DBA.VT_INDEX_DB_DBA_RDF_OBJ();
Thus far, no dice. I'm sort of wondering if it might have to do with the mangled characters in the comment field - the online dbpedia endpoint renders them properly, while my Virtuoso installation just gives question marks, as seen above. No idea even how to begin approaching this though.
I did include SQL_UTF8_EXECS = 1 in virtuoso.ini (and subsequently restart the server), which still left me with question marks in the results.
Actually, it does not appear to have anything to do with those question marks; I ran the following query:
select ?s, ?p, ?o where {
?s ?p ?o .
?o bif:contains "mythical" .
FILTER (!regex(?o, "mythical", "i"))
}
A pseudorandom selection of hits, none of which contain "mythical" or "?":
"Asgrrr"
"403 BC"
"Potential infinity"
"Beauty and the Beast (talk show)"
"Alberta highway highway 22"
The same query, run at http://dbpedia.org/sparql, returns nothing (as it should).
Any ideas?

Answer 1

Rebuilding the database did not correct the problem. I was, however, able to get a working version by taking the following steps. Some of them may have been unnecessary, but given how long it takes, I haven't done controlled experiments to narrow things down to the minimum.
First, delete the database and associated files, to start with a blank slate.
Edit virtuoso.ini to uncomment/include:
SQL_UTF8_EXECS = 1
Start up Virtuoso, and then from isql issue the following commands:
DB.DBA.RDF_OBJ_FT_RULE_ADD (null, null, 'All');
DB.DBA.VT_BATCH_UPDATE ('DB.DBA.RDF_OBJ', 'OFF', null);
DB.DBA.VT_INC_INDEX_DB_DBA_RDF_OBJ ();
DB.DBA.RDF_OBJ_FT_RECOVER ();
COMMIT WORK;
CHECKPOINT;
CHECKPOINT_INTERVAL(60000);
Then, load the data.
Then, call:
COMMIT WORK;
CHECKPOINT;
DB.DBA.VT_INC_INDEX_DB_DBA_RDF_OBJ();
CHECKPOINT;
COMMIT WORK;
CHECKPOINT;
CHECKPOINT_INTERVAL(60);
COMMIT WORK;
Enjoy your fully-text-searchable database!