ADO.net without writing SQL (esp. WHERE) - sql

I have this idea that using SQL VIEWS to abstract simple database computations (such as a count on a relation) is sufficient, and you don't need procedures (== procedural code)
A simple sql view + a where clause >> a stored procedure with parameters sometimes
While making this point I imagined a way of retrieving table/view data without writing SQL and without writing the where clause..
But, to my surprise, there does not seem a way to accomplish this in ADO.NET 2.0 or later.
Let me tell you what I tried:
SqlDataAdapter + SqlCommandBuilder still requires you to write "SELECT ... FROM" and the WHERE CLAUSE in strings (plus, if you put the 'where', you dont have much use of Update/Insert/DeleteCommand)
typed DataSets only allow you to retrieve _entire DataTable_s and then applying filters to them. Filters are strings, without escaping aid... (must double the single quote!)
SQL to Entities looked promising but they seem to: be limited to MSSQL, generate bloated SQL queries, generate a whole new stack of DAOs (besides the existing Domain Model classes), reqiuire .net 3.5+ for all this etc. (that is, all these are disadvantages for me)
Other ORMs have similar problems as SQL to Entities.
What I'm looking for is a strong-typed method of accessing database tables/views that:
doesn't come with another set of DAOs (K.I.S.S)
allows me to query a table without writing "SELECTs" in strings (strong-typed)
allows me to filter(WHERE) a table with properly-escaped parameters (and without retrieving the whole data beforehand)
can later issue updates/inserts/deletes
I am fairly new to .Net but not stupid: does this exist?
Thanks.

FluentADO

Subsonic has a fairly lightweight query tool that you can use to directly query against the database with a Query object that abstracts the SQL. If you want to, you can also use its code generation facility to map your database tables to POCOs, or to only create a strongly typed schema (for column/table names and so on).

I don't really believe that what you want to do is achievable without using some sort of ORM, or a specialized DSL with a compiler that somehow knows about your database schema, type/column information, etc.
Take into account that C# is a general purpose language, and its compiler is totally unaware of your database types, that's why you cannot bind them without using some abstraction layer, which usually involves ad hoc SQL queries(strings), NHibernate or similar mapping files (more strings) and/or DAOs.

If you don't want to write the WHERE clause, one way is to use a Filter object and add the conditions you want. For example:
var sc = new Filter();
sc.Add("Contacttitle", "Sales Agent");
sc.Add("city", "london", Logical.Or);
var customers = D2Bk.Fetch(sc, new Customers());
But you don't want to use DAOs (Customers above is such), so you would have to write the SQL statement and specify the where clause:
DataSet ds = D2Bk.Fetch("SELECT * FROM Customers WHERE Contacttitle=#PAR1 OR City=#PAR2", "Sales Agent", "london");

I would assume that you've looked at LINQ and ADO.Net Data Services and these do not meet some of your requirements?
Native ADO.Net is a database provider, and therefore it provides a direct SQL interface into the underlying data sources. There is various CRUB based solutions out there that simulate what you suggest to various degrees.
We have a strong internal trend to leave the database to the Database team and use Webservices as a main interface to our databases, mainly because of a Delphi codebase that still needs to be supported.
Actually can't believe I forgot to add ADO.Net Entity Framework which is used by ADO.Net Data Services among others. There is also a LINQ to Entities provider.

I did something like this with a stored procedure once. Basically, I wanted to specify any permutation of fields to match on in my WHERE clause, but I didn't want to write 100 sprocs with slightly different param lists and where clauses.
So, I did something like this:
CREATE PROCEDURE [GetSimpleCustomers]
(
#ID varchar(50) = null,
#Name varchar(50) = null,
#IsActive bit = null,
#Address1 varchar(50) = null,
#Address2 varchar(50) = null,
#City varchar(50) = null,
#State varchar(50) = null,
#Zip varchar(50) = null
)
AS
SELECT ID, Name, IsActive, Address1, Address2, City, State, Zip
FROM SimpleCustomerExample
WHERE (ID = #ID OR #ID is NULL)
AND (Name = #Name OR #Name is NULL)
AND (IsActive = #IsActive or #IsActive is NULL)
AND (Address1= #Address1 or #Address1 is NULL)
AND (Address2= #Address2 or #Address2 is NULL)
AND (City= #City or #City is NULL)
AND (State= #State or #State is NULL)
AND (Zip= #Zip or #Zip is NULL)
This will let you call the sproc in your code and only pass the params you are interested in filtering on, and the rest will not be factored in if you leave them null.
So, you can do something like
public List<SimpleCustomer> GetAllCustomersFromOhio()
{
List<SimpleCustomer> list = new List<SimpleCustomer>();
using (SqlCommand cmd = new SqlCommand(blah blah))
{
cmd.Parameters.AddWithValue("State", "Ohio");//or "OH" depending on your convention
using(IDataReader oDR = cmd.ExecuteReader())
{
//hydrate your list of SimpleCustomers from the record set.
}
}
return list;
}
EDIT:
In reply to comment:
You could easily enough alter the GetSimpleCustomers to be DeleteSimpleCustomers by changing the
SELECT <columns> FROM SimpleCustomers
to
DELETE FROM SimpleCustomers
and keep the same logic. The same is true for an Update.
Also, I'll answer a question with a question: How many tables do you have that actually need this level of custom filtering? The syntax would be so similar you could pound it all out in a day (or less if you hammered together a simple script to write it for you).

If you are using strongly-typed DataSets, you can create parameterized queries in the Visual Studio Editor by adding identifiers prefixed with # in the query. Create a DataSet XSD file in Visual Studio and create a new table called Products, then add a new query to it.
For example:
select * from Products where Category = #category;
This will autogenerate methods for filled datasets or getting datatables that take the extra parameter. It will also handle propery escaping the strings (uses parameters on the command objects). I used this to create some super simple prototype web apps fairly quickly.

I recently wrote a query 'framework' for generating SQL where clauses.
The primary component is a BaseQueryArgs class with a ToWhereClause() function that uses reflection to convert properties into string sections. This needs to handle the work of escaping and properly formatting values.
Any class inheriting BaseQueryArgs simply needs to declare public properties, and you end up with a strongly typed query object. For optional properties, you make the value nullable (ref type or Nullable<>) and the SQL generator filters out null values.
You can use custom attributes to define extra features for each property:
custom column name that differs from the property name
custom value handling (such as a date value being used as a BETWEEN test expression)
This can be used to create queries with a strongly-typed query object like so:
MyCustomQueryArgs args = new MyCustomQueryArgs
{
ProductFamilyID = 17,
Region = Regions.Northwest,
Active = true
};
List<Product> product = QueryProcessor.GetProductsWhere(args);
GetProductsWhere() would obviously call some data method that accesses the view with the generated SQL.
I don't have a solution for updates/deletes, but it doesn't seem that hard to write a method that converts an object instance into a SQL statement using a switch or attribute to determine table name.
This is very "roll your own", but that gives you the freedom to customize it for your needs, and doesn't include lots of heavy ORM/DAO wrapping.

Have a look at Mindscapes Lightspeed products
It builds stongly typed LINQ queriable models that results in efficient SQL code accross a variety of database engines and includes Memcached and Lucene support

I have used XPO on several projects and their newer version has better support for queries.
http://www.devexpress.com/Products/NET/ORM/
The implementation, like all of them is not without its drawbacks however.

I use Data Abstract for my projects.

Related

Query very fast but mapping slow with dapper

I'm using dapper for a new project and love it but I don't understand why my queries are really slow. The execution time is very fast, almost instant, but the connection stays open much longer while dapper is mapping the result to my object I guess.
Here is an example in glimpse :
This query is just a SELECT on something like 15 fields with a where on the primary key, so it's really fast to execute and it doesn't return that much of data.
My code to execute it is :
using (var conn = GetConnection())
{
obj = conn.Get<T>(id);
}
And the object is a very basic poco with Strings and Ints.
So why do I waste 220 ms doing this while the query execution itself takes 3 ms ? Where is the difference ?
Thanks for your help !
UPDATE
There was one field that was causing problems for me in the selection part of my SQL statement. I just went by removing each field one by one and then found the one which was causing the problem.
I had to cast one of my fields to nvarchar like this:
CAST(my_field AS nvarchar(max)) as my_field
ORIGINAL ANSWER
It has to do something with the mapping. Because if I change it from "Strongly Typed" (which is taking for ever, almost 1 minute):
var products = connection.Query<Product>(sql).ToList();
to "Anonymous":
var products = connection.Query(sql).ToList();
then it executes really fast (1 second).
I tried and executed the SQL statement directly in "SQL Server Management Studio" as a query and it finishes in less then 1 second.
So my suggestion is, that you use the "anonymous mapping" until dapper guys fix this if they will be able to.
I had a similar experience with Dapper as I was trying to project from a View to an POCO object.
The problem ended up being for me that I did not have a column for each property on my object, so the Convert.ChangeType() was very slow, I added a column to my View that would always return NULL, and the Query<T>() call sped up dramatically.
In my example, the database had an indexed column of type VARCHAR(10). I was attempting to filter via dapper parameter, like so:
DbConnection con = ...
string filterParam = "test";
var results = con.Query("SELECT IndexColumn, Column1, ... FROM MyTable WHERE IndexColumn = #filterParam", new { filterParam });
The issue was dapper (or possibly ADO.Net) converting my filterParam to NVARCHAR(MAX) data type. Sql Server then casts IndexColumn to NVARCHAR, and was doing a full table scan rather than indexed lookup. Code was fixed by casting the parameter before comparison:
var results = con.Query("SELECT IndexColumn, Column1, ... FROM MyTable WHERE IndexColumn = CAST(#filterParam AS VARCHAR(10))", new { filterParam });
In my case the poor performance seems to have been caused by the fact I was using an asterisk rather than a list of fields when doing the SELECT (i.e. SELECT * instead of SELECT Foo, Bar, Baz, ...).

suggestions on how to build query for search with multiple options

We are building a search form for users to search our database, the form will contain mulitlpe fields which are all optional. Fields including:
company name
company code
business type (service or product)
Product or Service
Product or Service subtype --> this will depend on what is chosen in #4
Basically the users can fill all or just some of the fields and submit the form. How best should we handle the sql for this? Is it best to use dynamic sql, build out the where clause in our webpage and then forward that to the sql stored procedure to use as it's where clause? Or is it better to pass all the values to the stored procedure and let it build the where clause dynamically. Also is dynamic sql the only way? I wasn't sure if using EXECUTE(#SQLStatement) is a good practice.
What I have done in the past is when a search option is not being usesd pass in a null for its value. Then in your select statement you would do something like
WHERE i.companyname = COALESCE(#CompanyName,i.companyname)
AND i.companycode = COALESCE(#CompanyCode,i.companycode)
What happens above is that if #CompanyName is null i.companyname will be compared to itself resulting in a match. If #CompanyName has a value it will compare i.companyname against that value.
I have used this way with 15 optional filters in a database with 15,000 rows and it has performed relatively well to date
More on the COALESCE operator
Dynamic SQL isn't the only way, it'd be better if you can avoid it with methods like: http://www.sommarskog.se/dyn-search.html
If you can't get the performance from the above method and go for dynamic SQL, do not allow the web-page to construct the SQL and execute it - you will end up getting SQL injected. Also avoid text strings being passed in, as sanitising them is very difficult. Ideally have the web page pass down parameters that are numbers only (IDs and such) for you to create the dynamic SQL from.
If you do decide to use dynamic SQL be sure to read all this: http://www.sommarskog.se/dynamic_sql.html

Wrap SQL CONTAINS as an expression?

I have a question. I working on one site on Asp.Net, which uses some ORM. I need to use a couple of FullTextSearch functions, such as Contains. But when I try to generate it with that ORM, it generates such SQL code
SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name]
FROM [dbo].[SomeTable] AS [Extent1]
WHERE (Contains([Extent1].[Name], N'qq')) = 1
SQL can't parse it, because Contains doesn't return bit value. And unfortunately I can't modify SQL query generation process, but I can modify statements in it.
My question is - is it possible to wrap call of CONTAINS function to something else? I tried to create another function, that will SELECT with contains, but it requires specific table\column objects, and I don't want to do one function for each table..
EDIT
I can modify result type for that function in ORM. In previous sample result type is Bit. I can change it to int,nvarchar,etc. But as I understood there is no Boolean type in SQL, and I can't specify it.
Can't you put this in a stored procedure, and tell your ORM to call the stored procedure? Then you don't have to worry about the fact that your ORM only understands a subset of valid T-SQL.
I don't know that I believe the argument that requiring new stored procedures is a blocker. If you have to write a new CONTAINS expression in your ORM code, how much different is it to wrap that expression in a CREATE PROCEDURE statement in a different window? If you want to do this purely in ORM, then you're going to have to put pressure on the vendor to pick up the pace and start getting more complete coverage of the language they should fully support.

VB.NET - Having some trouble with a parametized queries

I built a prototype system with some database queries. Since it was just a prototype and I was very new to databases, I just used a direct string. This is the string I used and it works fine:
command = New OleDbCommand("SELECT * FROM " + prefix + "CanonicForms WHERE Type=1 AND Canonic_Form='" + item + "'", dictionary_connection)
Now, in putting it in a real system, I wanted to use the more secure parametized method, so after some googling I came up with this:
command = New OleDbCommand("SELECT * FROM #prefix WHERE Type=1 AND Canonic_Form=#form", dictionary_connection)
command.Parameters.AddWithValue("#prefix", prefix + "CanonicForms")
command.Parameters.AddWithValue("#form", item)
But all I get is an error for an incomplete query clause. What have I done differently between the two?
Your table name can't be a parameter. You might have to do some form of concatenation. It's not really a parameter in the formal sense of the word.
As ek_ny states, your table name can not be a parameter.
If you are really paranoid about injection then check the passed in table name against a white list of allowable values prior to buidling up the SQL string.
#cost, I agree that it would be nice but its just not a feature that exists. Generally its assumed that your WHERE clauses are the dynamic portion of the query and everything else, including the SELECT list and the table name, are static. In fact, most of the WHERE clause isn't even dynamic, only the search values themselves. You can't dynamically add column names this way either. This could definitely be built but it would require that the query engine be more aware of the database engine which is a can of worms that Microsoft didn't feel like opening.
Imagine a drop down menu with table names such as 'Jobs', 'People', 'Employers' that some naive developer built. On postback that value is used to SELECT * FROM. A simple SQL injection would be all that it takes to jump to another table not listed in that menu. And passing something really weird could very easily throw an error that could reveal something about the database. Value-only parametrized queries can still be broken but the surface area is much smaller.
Some people with multiple table prefixes use schemas, is that an option for you?

SQL Server stored procedures - update column based on variable name..?

I have a data driven site with many stored procedures. What I want to eventually be able to do is to say something like:
For Each #variable in sproc inputs
UPDATE #TableName SET #variable.toString = #variable
Next
I would like it to be able to accept any number of arguments.
It will basically loop through all of the inputs and update the column with the name of the variable with the value of the variable - for example column "Name" would be updated with the value of #Name. I would like to basically have one stored procedure for updating and one for creating. However to do this I will need to be able to convert the actual name of a variable, not the value, to a string.
Question 1: Is it possible to do this in T-SQL, and if so how?
Question 2: Are there any major drawbacks to using something like this (like performance or CPU usage)?
I know if a value is not valid then it will only prevent the update involving that variable and any subsequent ones, but all the data is validated in the vb.net code anyway so will always be valid on submitting to the database, and I will ensure that only variables where the column exists are able to be submitted.
Many thanks in advance,
Regards,
Richard Clarke
Edit:
I know about using SQL strings and the risk of SQL injection attacks - I studied this a bit in my dissertation a few weeks ago.
Basically the website uses an object oriented architecture. There are many classes - for example Product - which have many "Attributes" (I created my own class called Attribute, which has properties such as DataField, Name and Value where DataField is used to get or update data, Name is displayed on the administration frontend when creating or updating a Product and the Value, which may be displayed on the customer frontend, is set by the administrator. DataField is the field I will be using in the "UPDATE Blah SET #Field = #Value".
I know this is probably confusing but its really complicated to explain - I have a really good understanding of the entire system in my head but I cant put it into words easily.
Basically the structure is set up such that no user will be able to change the value of DataField or Name, but they can change Value. I think if I were to use dynamic parameterised SQL strings there will therefore be no risk of SQL injection attacks.
I mean basically loop through all the attributes so that it ends up like:
UPDATE Products SET [Name] = '#Name', Description = '#Description', Display = #Display
Then loop through all the attributes again and add the parameter values - this will have the same effect as using stored procedures, right??
I dont mind adding to the page load time since this is mainly going to affect the administration frontend, and will marginly affect the customer frontend.
Question 1: you must use dynamic SQL - construct your update statement as a string, and run it with the EXEC command.
Question 2: yes there are - SQL injection attacks, risk of mal-formed queries, added overhead of having to compile a separate SQL statement.
Your example is very inefficient, so if I pass in 10 columns you will update the same table 10 times?
The better way is to do one update by using sp_executesql and build this dynamically, take a look at The Curse and Blessings of Dynamic SQL to see how you have to do it
Is this a new system where you have the freedom to design as necessary, or are you stuck with an existing DB design?
You might consider representing the attributes not as columns, but as rows in a child table.
In the parent MyObject you'd just have header-level data, things that are common to all objects in the system (maybe just an identifier). In the child table MyObjectAttribute you'd have a primary key of with another column attrValue. This way you can do an UPDATE like so:
UPDATE MyObjectAttribute
SET attrValue = #myValue
WHERE objectID = #myID
AND attrName = #myAttrName