SQL Select/From/Where Run Speed - sql

Question

I have a program that is pulling data from a Visual FoxPro table and dumping into a Dataset with VB.net. My connection string works great, and the query I'm using usually runs with respectable speed. As I've ran it more, however, I've learned that there is a large amount of "bad" data in my table. So now, I'm trying to refine my query to buffer against the "bad" data, but what I thought would be a very small tweak has yielded massive performance losses, and I'm not particularly sure why.
My original query is:
'Pull desired columns for orders that have not "shipped" and were received in past 60 days.
'To "ship", an order must qualify with both an updated ship date and Sales Order #.
sqlSelect = "SELECT job_id,cust_id,total_sale,received,due,end_qty,job_descr,shipped,so "
sqlFrom = "FROM job "
sqlWhere = "WHERE fac = 'North Side' AND shipped < {12/30/1899} AND so = '' AND received >= DATE()-60;"
sql = sqlSelect & sqlFrom & sqlWhere
This has a run-time of about 20 seconds; while I'd prefer it to be quicker, it's not a problem. In my original testing (and occasional debugging), I replaced sqlWhere with sqlWhere = "WHERE job_id = 127350". This runs pretty much instantaneously.
Now the problem block: Once I replaced sqlWhere with
'Find jobs that haven't "shipped" OR were received within last 21 days.
'Recently shipped items are desired in results.
sqlWhere = "WHERE fac = 'North Side' AND ((shipped < {12/30/1899} AND so = '') OR received >= DATE()-21);"
My performance jumped to about 3 min 40 sec. This time is almost exactly the same as the time to run with sqlWhere = "WHERE received >= DATE();".
I'm not the moderator of these tables; I'm merely pulling from them to create a series of reports for our users. My best guess is that the received field is not indexed, this is the cause of my performance drop-off. But while my first search returns about 100 records, pulling the jobs only from today returns about 5, and still takes about 11x as long.
So my question is three part:
1) Would someone be able to explain the phenomenon I'm experiencing right now? I feel like I'm somewhat on the right track, but my knowledge of SQL has been limited to circumstantial use within other languages...
2) Is there something I'm missing, or some better way to obtain the results I need? There are a large volume of records that haven't "shipped", but simply because the user only input a shipped date or s/o, and didn't do the other. I need a way to view very recent orders (regardless of "shipped" status), and then also view less recent orders that have "bad" data, so I can get the user in the habit of cleaning up the data.
3) Is it bad SQL practice to overconstrain a WHERE clause? If I run fifteen field comparisons, joined together with nested ANDs/ORs, am I wasting my time when I could be doing something much cleaner?
Many thanks,
B

Answer 1

If you are looking for a non-indexed record in your WHERE string, the SQL engine must do a table scan, i.e. - look at every record in the table.
The difference between the two queries is having the OR instead of the AND. When you have a non-indexed column in an AND, the SQL engine can use the indexes to narrow down the number of records it has to look at for the non-indexed column. When you have an OR, it now must look at every record in the table and compare on that column.
Adding an index on the Received column would probably fix the performance issue.
In general, there are two things you don't want to have happen in your WHERE clause.
1. A primary condition on an non-indexed column
2. Using a calculation on a column. For example, doing WHERE Shipped-2 < date() is often worse than doing Shipped < Date() + 2, because the former doesn't typically allow the index to be used.
Refining your query through multiple WHERE clauses is generally a good thing. The fewer records you need to return to your application the better your performance will be, but you need to have appropriate indexing in place.