Following the question at Difference in performance between using VALUES keyword and using directly the URI in the query?, I learned that using a VALUES clause at the end of the query is not always equivalent in terms of performance and query optimization than using directly a URI instead of the variable in the query string.
Comment from Andy says "VALUES at the end is "like setting variables" but isn't the same. The optimizer tries to push the values in but that can't happen in all cases as it changes the semantics."
Can someone explain in which cases this can't happen? for which query structures, and why exactly? I need to understand in which situations this technique (that I happily used for years now) is not advisable.
Note that I am not fluent with SPARQL algebra, so please try using simple words :-)
(I know this is not specific to Jena or RDF4J but I tag the question with these 2 tags since I understood the optimization of this might be different depending on the framework used).
Related
My client is making database searches using a django webapp that I've written. The query sends a regex search to the database and outputs the results.
Because the regex searches can be pretty long and unintuitive, the client has asked for certain custom "wildcards" to be created for the regex searches. For example.
Ω := [^aeiou] (all non-vowels)
etc.
This could be achieved with a simple permanent string substitution in the query, something like
query = query.replace("Ω", "[^aeiou]")
for all the elements in the substitution list. This seems like it should be safe, but I'm not really sure.
He has also asked that it be possible for the user to define custom wildcards for their searches on the fly. So that there would be some other input box where a user could define
∫ := some other regex
And to store them you might create a model
class RegexWildcard(models.Model):
symbol = ...
replacement = ...
I'm personally a bit wary of this, because it does not seem to add a whole lot of functionality, but does seem to add a lot of complexity and potential problems to the code. Clients can now write their queries to a db. Can they overwrite each other's symbols?
That I haven't seen this done anywhere before also makes me kind of wary of the idea.
Is this possible? Desirable? A great idea? A terrible idea? Resources and any guidance appreciated.
Well, you're getting paid by the hour....
I don't see how involving the Greek alphabet is to anyone's advantage. If the queries are stored anywhere, everyone approaching the system would have to learn the new syntax to understand them. Plus, there's the problem of how to type the special symbols.
If the client creates complex regular expressions they'd like to be able to reuse, that's understandable. Your application could maintain a list of such expressions that the user could add to and choose from. Notionally, the user would "click on" an expression, and it would be inserted into the query.
The saved expressions could have user-defined names, to make them easier to remember and refer to. And you could define a syntax that referenced them, something otherwise invalid in SQL, such as ::name. Before submitting the query to the DBMS, you substitute the regex for the name.
You still have the problem of choosing good names, and training.
To prevent malformed SQL, I imagine you'll want to ensure the regex is valid. You wouldn't want your system to store a ; drop table CUSTOMERS; as a "regular expression"! You'll either have to validate the expression or, if you can, treat the regex as data in a parameterized query.
The real question to me, though, is why you're in the vicinity of standardized regex queries. That need suggests a database design issue: it suggests the column being queried is composed of composite data, and should be represented as multiple columns that can be queried directly, without using regular expressions.
Question:
Is there a way to use names instead of question marks for paramaterized queries? If so, can anyone suggest some material that explains how to do this/the syntax?
A bit more detail:
For example, if I have something like:
INSERT INTO inventory VALUES(?)
Is it possible to have something like this instead that does the exact same thing as the question mark:
INSERT INTO inventory VALUES("prices")
I tried checking to see if it would work myself before posting the question, but it didn't work. So, I thought I'd ask if it was possible.
I feel like if you have a really long query with, let's say 20 parameters, you don't want to have to count question marks to make sure you have enough parameters whenever you change something. Also, I think it might make the code a bit more readable (especially if you have a lot of parameters to keep track of).
I'm rather new to sql, so I am not sure if it makes much of a difference (for this question) if I add that I'm using postgresql.
Note:
There is a similar question here, but it didn't have an answer that was helpful
I suggest to encapsulate the big query in a function, where you can use parameter names.
One example (out of many):
PostgreSQL parameterized Order By / Limit in table function
You can even set default values and call the function with named parameters, etc.:
Functions with variable number of input parameters
I've been working off of the tutorial pages but seem to have a fundamental disconnect in my thinking transitioning off of RDBMS systems. I'm using MarkLogic and handling this database interaction through the Java API focusing on the search access via POJO method outlines in the tutorial documentation.
My reference up to this point has come from here principally: http://developer.marklogic.com/learn/java/processing-search-results
My scenario is this:
I have a series of documents. We'll call them 'books' for simplicity. I'm writing these books into my DB like this:
jsonDocMgr.write("/" + book.getID() + "/",
new StringHandle(
"{name: \""+book.getID()+"\","+
"chaps: "+ book.getNumChaps()+","+
"pages: "+ book.getNumPages()+","+
"}"));
What I want is to execute the following type of operation:
-Query all documents with the name "book*" (as ID is represented by book0, book1, book2, etc)
where chaps > 3. For these documents only, I want to modify the number of pages by reducing by half.
In an RDBMS, I'd use something like jdbcTemplate and get a result set for me to iterate through. For each iteration I'd know I was working with a single record (aka a book), parse the field values from the result set, make a note of the ID, then update the DB accordingly.
With MarkLogic, I'm awash in a sea of different handlers and managers...none of which seems to follow the pattern of the ResultSet with a cursor abstraction. Ultimately I want to do a two-step operation of check the chapter count then update the page field for that specific URI.
What's the most common approach to this? It seems like the most basic of operations...
Try the high-level Java API and see if it works for you. Create a multi-statement transaction with a query by example, then use document operations.
At a lower level, the closest match to a ResultSet is the ResultSequence class. The examples at http://docs.marklogic.com/javadoc/xcc/overview-summary.html are pretty good. For updates the interaction model between Java and MarkLogic is a bit different from JDBC and SQL. There is no SELECT... FOR UPDATE syntax.
The most efficient low-level technique is to select and update in one XQuery transaction, something like a stored procedure. However this requires good knowledge of XQuery. The other low-level approach is to use an XCC multi-statement transaction, which requires a little less knowledge of XQuery.
A minor issue in your code ... you definately do NOT want to end your JSON docuement URIs with "/" as you do in your sample code. You should end them with the ".json" or some other extension or no extension but definately not "/" as that is treated specially in the server.
I would like to build a safe dynamic select statement that can handle multiple WHERE clauses.
For example the base SQL would look like:
SELECT * FROM Books Where Type='Novel'
I would pass the function something like:
SafeDynamicSQL("author,=,Herman Melville","pages,>,1000");
Which would sanitize inputs and concatenate like:
SELECT * FROM Books Where Type='Novel' AND author=#author AND pages>#pages
The function would sanitize the column name by checking against an array of predefined column names. The operator would only be allowed to be >,<,=. The value would be added as a normal paramater.
Would this still be vulnerable to SQL injection?
There will be some string manipulation and small loops which will affect performance but my thoughts are that this will only take a few milliseconds compared to the request which on average take 200ms. Would this tax the server more than I am thinking if these requests are made about once a second?
I know this isn't best practice by any means, but it will greatly speed up development. Please give me any other reasons why this could be a bad idea.
It looks like you're reinventing any number of existing ORM solutions which offer a similar API for creating WHERE clauses.
The answer to your question hinges on what you mean by "The value would be added as a normal paramater." If by that you mean performing string concatenation to produce the string you showed then yes, that would still be subject to SQL injection attack. If you mean using an actual parameterized query then you would be safe. In the later case, you would produce something like
SELECT * FROM Books Where Type='Novel' AND author=? AND pages > ?
and then bind that to a list of values like ['Herman Melville', 1000]. Exactly what it would look like depends on what programming language you're using.
Finally, if you pursue this path I would strongly recommend changing from comma-delimited arguments to three separate arguments, you'd save yourself a lot of programming time.
Pretty much any code that appends together (or interpolates) strings to create SQL is bad form from a security point of view, and is probably subject to some SQLi attack vector. Just use bound parameters and avoid the entire problem; avoiding SQL injection is super-easy.
I was writing some Unit tests last week for a piece of code that generated some SQL statements.
I was trying to figure out a regex to match SELECT, INSERT and UPDATE syntax so I could verify that my methods were generating valid SQL, and after 3-4 hours of searching and messing around with various regex editors I gave up.
I managed to get partial matches but because a section in quotes can contain any characters it quickly expands to match the whole statement.
Any help would be appreciated, I'm not very good with regular expressions but I'd like to learn more about them.
By the way it's C# RegEx that I'm after.
Clarification
I don't want to need access to a database as this is part of a Unit test and I don't wan't to have to maintain a database to test my code. which may live longer than the project.
Regular expressions can match languages only a finite state automaton can parse, which is very limited, whereas SQL is a syntax. It can be demonstrated you can't validate SQL with a regex. So, you can stop trying.
SQL is a type-2 grammar, it is too powerful to be described by regular expressions. It's the same as if you decided to generate C# code and then validate it without invoking a compiler. Database engine in general is too complex to be easily stubbed.
That said, you may try ANTLR's SQL grammars.
As far as I know this is beyond regex and your getting close to the dark arts of BnF and compilers.
http://savage.net.au/SQL/
Same things happens to people who want to do correct syntax highlighting. You start cramming things into regex and then you end up writing a compiler...
I had the same problem - an approach that would work for all the more standard sql statements would be to spin up an in-memory Sqlite database and issue the query against it, if you get back a "table does not exist" error, then your query parsed properly.
Off the top of my head: Couldn't you pass the generated SQL to a database and use EXPLAIN on them and catch any exceptions which would indicate poorly formed SQL?
Have you tried the lazy selectors. Rather than match as much as possible, they match as little as possible which is probably what you need for quotes.
To validate the queries, just run them with SET NOEXEC ON, that is how Entreprise Manager does it when you parse a query without executing it.
Besides if you are using regex to validate sql queries, you can be almost certain that you will miss some corner cases, or that the query is not valid from other reasons, even if it's syntactically correct.
I suggest creating a database with the same schema, possibly using an embedded sql engine, and passing the sql to that.
I don't think that you even need to have the schema created to be able to validate the statement, because the system will not try to resolve object_name etc until it has successfully parsed the statement.
With Oracle as an example, you would certainly get an error if you did:
select * from non_existant_table;
In this case, "ORA-00942: table or view does not exist".
However if you execute:
select * frm non_existant_table;
Then you'll get a syntax error, "ORA-00923: FROM keyword not found where expected".
It ought to be possible to classify errors into syntax parsing errors that indicate incorrect syntax and errors relating to tables name and permissions etc..
Add to that the problem of different RDBMSs and even different versions allowing different syntaxes and I think you really have to go to the db engine for this task.
There are ANTLR grammars to parse SQL. It's really a better idea to use an in memory database or a very lightweight database such as sqlite. It seems wasteful to me to test whether the SQL is valid from a parsing standpoint, and much more useful to check the table and column names and the specifics of your query.
The best way is to validate the parameters used to create the query, rather than the query itself. A function that receives the variables can check the length of the strings, valid numbers, valid emails or whatever. You can use regular expressions to do this validations.
public bool IsValid(string sql)
{
string pattern = #"SELECT\s.*FROM\s.*WHERE\s.*";
Regex rgx = new Regex(pattern, RegexOptions.IgnoreCase);
return rgx.IsMatch(sql);
}
I am assuming you did something like .\* try instead [^"]* that will keep you from eating the whole line. It still will give false positives on cases where you have \ inside your strings.