Entity SQL Without Metadata? - sql

I have a need to generate SQL queries against a variety of database providers, for database schemas that are not known at compile time. I see that Entity Framework has already done the hard work of providing a SQL dialect called Entity SQL that is translated to native SQL before execution, and I was hoping to take advantage of that in some way.
Ideally, I'd like to just generate ESQL, run it, and get an IDataReader, while Entity Framework worries about the provider-specific details. However, there doesn't seem to be a way to create an EntityConnection without providing metadata in the form of SSDL, CSDL, and MSL files, and I won't find out about the database schema until runtime.
My question is, is there any way to take advantage of ESQL without any information about the database schema at compile time? If necessary (and possible), I would be open to programmatically generating the metadata from the database and caching it. I would also be open to any .NET tools that might be a better fit for my needs than Entity Framework.
Thanks for your time.
Update
Thanks to Alex's suggestion, I've been able to work out how to generate the metadata needed for an EntityConnection on the fly, without writing out any files. As a result, I've been able to do exactly what I was hoping. Now all I need to do is figure out how to extract information about available tables/views for my own use from the generated metadata.
Here's my test code:
#r "System.Data.Entity"
#r "System.Data.Entity.Design"
#r "System.Transactions"
open System
open System.IO
open System.Data
open System.Data.EntityClient
open System.Data.Entity.Design
open System.Data.Mapping
open System.Data.Metadata.Edm
open System.Data.SqlClient
open System.Text
open System.Xml
let dbName = "Northwind"
let cnstr = sprintf "Server=.;Database=%s;Integrated Security=SSPI" dbName
let provider = "System.Data.SqlClient"
let mslText = StringBuilder()
let mslWriter = XmlWriter.Create(mslText)
let schemaGen = EntityStoreSchemaGenerator(provider, cnstr, dbName)
schemaGen.GenerateStoreMetadata() |> ignore
let modelGen = EntityModelSchemaGenerator(schemaGen.EntityContainer)
modelGen.GenerateMetadata() |> ignore
modelGen.WriteStorageMapping(mslWriter)
mslWriter.Close()
let mslReader = XmlReader.Create(new StringReader(mslText.ToString()))
let ssdlCollection = schemaGen.StoreItemCollection
let csdlCollection = modelGen.EdmItemCollection
let mslCollection = StorageMappingItemCollection(csdlCollection, ssdlCollection, [| mslReader |])
let mdw = MetadataWorkspace()
mdw.RegisterItemCollection(csdlCollection)
mdw.RegisterItemCollection(ssdlCollection)
mdw.RegisterItemCollection(mslCollection)
let sqlCn = new SqlConnection(cnstr)
let cn = new EntityConnection(mdw, sqlCn)
let cmd = cn.CreateCommand()
cmd.CommandText <- sprintf "SELECT p.ProductName FROM %sContext.Products AS p" dbName
cn.Open()
let reader = cmd.ExecuteReader(CommandBehavior.SequentialAccess)
while reader.Read() do
printfn "%A" reader.["ProductName"]
reader.Close()
cn.Close()
sqlCn.Close()

There is a way to create an EntityConnection by pointing at appropriate metadata (CSDL, MSL and SSDL) at runtime.
In fact that is one of the key benefits of dropping down from the Object Layer (CLR classes) to the Entity Client (eSQL and DataReaders etc).
To help you get started this This post shows you how to create an EntityConnection at runtime.

Related

Linked SQL table in Access 2003 (!) not updatable

I'm working in a legacy app for the moment, upgrading Access 2003 to link to SQL Server tables (2008 R2 or later). With tables linked by code, I can insert, but not update or delete. I've tried everything on the web, no dice. Details below.
Being terse so not tl;dr.
Tables first created using upsizing wizard. In use, app has to connect to different ones in same schema, so can't just set and forget. Can't do local DSN's, many installs, though DSN file is possible. But problems there too, DSN not found. Details later.
Before the rest: Soon I'm further updating this app to Access 2016 or so. If this is different enough / easier there, I'll wait a few days. Maybe someone could suggest the best refsite for that.
* problem details follow *
Using a DSN and the UI to link a table, I get an editable table. Hurray.
But when I use the code below (found on every refsite), link is made but only selecting and inserting work. Everything else fails fails fails, no matter what.
Public Function LinkToSqlTable(sqlInstance As String, sqlDb As String,
sqlTableName As String, localTableName As String)
Dim linked As New TableDef
' ***factored-out functionality, known to work: reader can ignore*** '
DeleteTable localTableName
' connection-string steps, placeholders replaced by args '
Dim sCnx As String
sCnx = "ODBC;Driver=SQL Server;Server=_instance_;" & _
"Database=_db_;Integrated Security=SSPI"
sCnx = Replace(sCnx, "_instance_", sqlInstance)
sCnx = Replace(sCnx, "_db_", sqlDb)
' linked-table steps '
Set linked = CurrentDb.CreateTableDef(localTableName)
linked.Connect = sCnx
linked.SourceTableName = sqlTableName
CurrentDb.TableDefs.Append linked
' ui '
RefreshDatabaseWindow
End Function
* ID column or permissions? *
I thought the problem was lack of identity column originally, I added one, but no change. At least now I have a PK field like I should. ;-)
When I manually link table, UI demands to know the ID column. So could it still be it? Fine, but how do I set that in code? Searches revealed nothing.
I assume then it's permissions as sites etc. say. I also took all the steps I could think of to fix that. No dice.
* things I've tried *
Aside from the ID-column stuff I said before, these things (not in order):
Since DSN saved as a file, tried using it as exampled, in cnx string. Fail.
Used DSN contents, carefully winnowed & translated, in cnx string. Fail.
Used connection string from the table that I had connected manually with DSN. Fail.
Changed driver in cnx string across all major options, even omitted it. Fail.
Changed security in cnx to Integrated Security=SSPI and other options, and omitted entirely. Fail.
I added my actual local user as exampled, with and without password. Fail.
(Previous few options tried across earlier options, though not 100% coverage.)
In SQL Server, using SSMS, I tried security power:
Added SQS-authentication login to the instance
Matching user to the default db seen here
Gave that login-user read and write permissions in db here (plus others, sometimes)
Added matching id & pw to the cnx string. Fail.
I tried setting up this db in SQS to have let-everyone-do-everything "security" temporarily. Fail.
This, that, and the other thing. Everything fail!!
So a permissions issue? Some way to use DSN file after all? Mismatched permission settings in my cnx string? Boneheaded oversight? Something else that I've missed? I'm pretty good at both SQL Server and Access, but only at a basic level in their security stuff and connection strings are the devil.
* retrieved table properties *
Just in case they help, I retrieved these (after objects added to TableDefs collection).
** This one, done in UI and with DSN and this-is-ID-field, worked with editing: **
Name = dbo_tblSendTo
Updatable = False
DateCreated = 4/19/2016 11:11:40 AM
LastUpdated = 4/19/2016 11:11:42 AM
Connect = ODBC;Description=SQL Server tables for TeleSales 5;DRIVER=SQL Server Native Client 10.0;SERVER=(local)\sqlexpress;Trusted_Connection=Yes;APP=Microsoft Office 2003;WSID=CMSERVER;DATABASE=TS5_General;
Attributes = 536870912
SourceTableName = dbo.tblSendTo
RecordCount = -1
ValidationRule =
ValidationText =
ConflictTable =
ReplicaFilter =
** And this one, from table linked via code, didn't: **
Name = tblSendTo
Updatable = False
DateCreated = 4/19/2016 11:17:51 AM
LastUpdated = 4/19/2016 11:17:51 AM
Connect = ODBC;Description=SQL Server tables for TeleSales 5;DRIVER=SQL Server Native Client
> 10.0;SERVER=(local)\sqlexpress;Trusted_Connection=Yes;APP=Microsoft Office 2003;WSID=CMSERVER;DATABASE=TS5_General;
Attributes = 536870912
SourceTableName = dbo.tblSendTo
RecordCount = -1
ValidationRule =
ValidationText =
ConflictTable =
ReplicaFilter =
* my plea *
So..... Please someone help me out. I don't like feeling stupid like this, and regrettably I need to do this instead of replacing it with .NET code or similar.
Thanks, anyone who can...
Ed.
Alas, I am able to answer my own question.
edited a little since first posted in reply to HansUp's comments
I had added an identity column to the table that I couldn't edit. However, I had not set it up as a primary key. It turns out that using identity doesn't make something a primary key automatically.
But the latter, making it primary key using either of the 2 possible DDL syntaxes, is crucial. Since I thought I had dealt with the no edits without unique key problem, I focused on permissions.
All of the permissions things here, then, are just a sideshow.
The upshot of this is to be sure to add an identity column and make it a primary key if for some reason your original table schema didn't have that.
If I have the time, I will be trimming the question to reflect what I've discovered.

Easy way to get and modify a .vb file like a text file

I'm creating a vb.net application for converting a Data Access Layer classes from Oracle DB oriented to SQL Server DB oriented ,
And i must convert every Oracle oriented class like :
OracleCommand , OracleDataAdapter , OracleDataReader ...
to it's SQL Server equivalent .
Is there any way to get methods list from .vb file and to get the body of each method from vb application to change it dynamically and create new version of each file ???
If you compile in debug mode, one of the output files is PROGRAMNAME.xml, which contains all the method signatures for the EXE. (Where PROGRAMNAME is the name of your EXE).
You can write a program to edit the .vb files just like any other text file, because they are text files.
Download trial version of Resharper and it will help you to do it with no pain. One month trial should be enough.
VS also has some tools - "Find all references" and "Find and Replace in Files". So, you can Find and replace Oracle... To Sql...
On the other note, why not writing wrappers that will provide DB-agnostic providers for command, transaction, etc. For example, instead of
Dim reader As OracleDataReader = OracleCommand.ExecuteReader...
Do this
Dim reader As IDataReader = MyAgnosticCommand.ExecuteReader...
So, if the new boss is coming tomorrow and says, "Oh we need oracle" - you don't have to do it all over again.

Trouble establishing connection to Local SQL database

Simply trying to find out the correct syntax for my connection string. Before anyone asks, yes I did look at other SO answers, and no they did not work for me. Here a couple of attempts I made from looking at other SO questions like the one I am asking
Server=(local);Database=SalesOrdersExample;Integrated Security= true
Data Source=(local);Database=SalesOrdersExample;Integrated Security=SSPI
Server=.\\SQLEXPRESS;Database=SalesOrdersExampleDataSet;Integrated Security=true
None of them worked (I have a Console.WriteLine("test"); thrown in there and it works up until I try conn.Open() (opening the connection to database) so I'm assuming that it must be my connection string since nothing gets written after conn.Open())
Console.WriteLine("test"); // works
SqlConnection conn = new SqlConnection("Server=.\\SQLEXPRESS;Database=SalesOrdersExampleDataSet;Integrated Security=true");
Console.WriteLine("test"); // works
conn.Open();
Console.WriteLine("test"); // does not work
So some information about the database is that it's local under my 'Data Connections' in my Server Explorer. I also have the .xsd file in my project so I have linked the Data Set to the current project I am on. Here is a picture representation to confirm that I have both the Data Connection and the Data Set.
EDIT: SO does not allow me to post pictures until I have 10 rep so here is direct link to picture:
DB Screenshot
Any help is appreciated thank you.
Visual Studio comes with LocalDB database, which is not exactly SQL Server Express database.
Try something like this:
Server=(localdb)\v11.0;Integrated Security=true;
or
Data Source=(LocalDB)\v11.0; AttachDbFileName=|DataDirectory|\DatabaseFileName.mdf; InitialCatalog=DatabaseName;Integrated Security=True;MultipleActiveResultSets=True
If using in c# code, you can use # to avoid problems with backslash characters:
SqlConnection conn =
new SqlConnection(#"Server=(localdb)\v11.0;Integrated Security=true;");

Create a sqllte file with password [duplicate]

I'm just learning to use SQLite and I was curious if such is possible:
Encryption of the database file?
Password protect opening of the database?
PS. I know that there is this "SQLite Encryption Extension (SEE).", but according to the documentation, "The SEE is licensed software...." and "The cost of a perpetual source code license for SEE is US $2000."
SQLite has hooks built-in for encryption which are not used in the normal distribution, but here are a few implementations I know of:
SEE - The official implementation.
wxSQLite - A wxWidgets style C++ wrapper that also implements SQLite's encryption.
SQLCipher - Uses openSSL's libcrypto to implement.
SQLiteCrypt - Custom implementation, modified API.
botansqlite3 - botansqlite3 is an encryption codec for SQLite3 that can use any algorithms in Botan for encryption.
sqleet - another encryption implementation, using ChaCha20/Poly1305 primitives. Note that wxSQLite mentioned above can use this as a crypto provider.
The SEE and SQLiteCrypt require the purchase of a license.
Disclosure: I created botansqlite3.
You can password protect SQLite3 DB.
For the first time before doing any operations, set password as follows.
SQLiteConnection conn = new SQLiteConnection("Data Source=MyDatabase.sqlite;Version=3;");
conn.SetPassword("password");
conn.open();
then next time you can access it like
conn = new SQLiteConnection("Data Source=MyDatabase.sqlite;Version=3;Password=password;");
conn.Open();
This wont allow any GUI editor to view Your data.
Later if you wish to change the password, use conn.ChangePassword("new_password");
To reset or remove password, use conn.ChangePassword(String.Empty);
The .net library System.Data.SQLite also provides for encryption.
You can get sqlite3.dll file with encryption support from http://system.data.sqlite.org/.
1 - Go to http://system.data.sqlite.org/index.html/doc/trunk/www/downloads.wiki and download one of the packages. .NET version is irrelevant here.
2 - Extract SQLite.Interop.dll from package and rename it to sqlite3.dll. This DLL supports encryption via plaintext passwords or encryption keys.
The mentioned file is native and does NOT require .NET framework. It might need Visual C++ Runtime depending on the package you have downloaded.
UPDATE
This is the package that I've downloaded for 32-bit development: http://system.data.sqlite.org/blobs/1.0.94.0/sqlite-netFx40-static-binary-Win32-2010-1.0.94.0.zip
Keep in mind, the following is not intended to be a substitute for a proper security solution.
After playing around with this for four days, I've put together a solution using only the open source System.Data.SQLite package from NuGet. I don't know how much protection this provides. I'm only using it for my own course of study. This will create the DB, encrypt it, create a table, and add data.
using System.Data.SQLite;
namespace EncryptDB
{
class Program
{
static void Main(string[] args)
{
string connectionString = #"C:\Programming\sqlite3\db.db";
string passwordString = "password";
byte[] passwordBytes = GetBytes(passwordString);
SQLiteConnection.CreateFile(connectionString);
SQLiteConnection conn = new SQLiteConnection("Data Source=" + connectionString + ";Version=3;");
conn.SetPassword(passwordBytes);
conn.Open();
SQLiteCommand sqlCmd = new SQLiteCommand("CREATE TABLE data(filename TEXT, filepath TEXT, filelength INTEGER, directory TEXT)", conn);
sqlCmd.ExecuteNonQuery();
sqlCmd = new SQLiteCommand("INSERT INTO data VALUES('name', 'path', 200, 'dir')", conn);
sqlCmd.ExecuteNonQuery();
conn.Close();
}
static byte[] GetBytes(string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
bytes = System.Text.Encoding.Default.GetBytes(str);
return bytes;
}
}
}
Optionally, you can remove conn.SetPassword(passwordBytes);, and replace it with conn.ChangePassword("password"); which needs to be placed after conn.Open(); instead of before. Then you won't need the GetBytes method.
To decrypt, it's just a matter of putting the password in your connection string before the call to open.
string filename = #"C:\Programming\sqlite3\db.db";
string passwordString = "password";
SQLiteConnection conn = new SQLiteConnection("Data Source=" + filename + ";Version=3;Password=" + passwordString + ";");
conn.Open();
You can always encrypt data on the client side. Please note that not all of the data have to be encrypted because it has a performance issue.
You can use SQLite's function creation routines (PHP manual):
$db_obj->sqliteCreateFunction('Encrypt', 'MyEncryptFunction', 2);
$db_obj->sqliteCreateFunction('Decrypt', 'MyDecryptFunction', 2);
When inserting data, you can use the encryption function directly and INSERT the encrypted data or you can use the custom function and pass unencrypted data:
$insert_obj = $db_obj->prepare('INSERT INTO table (Clear, Encrypted) ' .
'VALUES (:clear, Encrypt(:data, "' . $passwordhash_str . '"))');
When retrieving data, you can also use SQL search functionality:
$select_obj = $db_obj->prepare('SELECT Clear, ' .
'Decrypt(Encrypted, "' . $passwordhash_str . '") AS PlainText FROM table ' .
'WHERE PlainText LIKE :searchterm');
Well, SEE is expensive. However SQLite has interface built-in for encryption (Pager). This means, that on top of existing code one can easily develop some encryption mechanism, does not have to be AES. Anything really.
Please see my post here: https://stackoverflow.com/a/49161716/9418360
You need to define SQLITE_HAS_CODEC=1 to enable Pager encryption. Sample code below (original SQLite source):
#ifdef SQLITE_HAS_CODEC
/*
** This function is called by the wal module when writing page content
** into the log file.
**
** This function returns a pointer to a buffer containing the encrypted
** page content. If a malloc fails, this function may return NULL.
*/
SQLITE_PRIVATE void *sqlite3PagerCodec(PgHdr *pPg){
void *aData = 0;
CODEC2(pPg->pPager, pPg->pData, pPg->pgno, 6, return 0, aData);
return aData;
}
#endif
There is a commercial version in C language for SQLite encryption using AES256 - it can also work with PHP, but it needs to be compiled with PHP and SQLite extension. It de/encrypts SQLite database file on the fly, file contents are always encrypted. Very useful.
http://www.iqx7.com/products/sqlite-encryption
I had also similar problem. Needed to store sensitive data in simple database (SQLite was the perfect choice except security). Finally I have placed database file on TrueCrypt encrypted valume.
Additional console app mounts temporary drive using TrueCrypt CLI and then starts the database application. Waits until the database application exits and then dismounts the drive again.
Maybe not suitable solution in all scenarios but for me working well ...

How can I use LinqPad's generated context in Visual Studio

This is a follow-on from this question really:
Moving From LINQpad to a Proper Visual Studio Project?
..but I'm not able to get it to work properly.
An answer to that question suggestions dumping the context assembly out as a dll but although I have done that, when I import it as a reference, it's not exactly clear to me how I would create an instance of that context, point it at a database and actually run a query against it, something like the following:
var db = new ContextFromThatDLL(myconnectionstring);
var query = from a in db.MYTABLE where a.ID == 1 select a;
Extra information:
I am using the IQ driver in LinqPad to connect to Oracle.
I do have a license for DevArt already (which the IQ driver uses) but am aware that the IQ driver generates its own SQL from LINQ - and I prefer it. Plus, I develop queries in LinqPad which works great for my workflow but find that DevArt doesn't always generate SQL as good as IQ.
First, extract the typed data context in LINQPad as follows:
string dcPath = GetType().BaseType.Assembly.Location;
string targetFolder = #"c:\temp";
File.Copy (dcPath, Path.Combine (targetFolder, Path.GetFileName (dcPath)));
Then in Visual Studio, reference the typed data context DLL, along with the following DLLs from the driver folder:
IQDriver.dll
IQToolkit.dll
IQToolkit.Data.dll
IQToolkit.Data.(provider).dll
plus the DevArt driver.
Then, you can instantiate the typed data context as follows (this illustrates how to do it for SQLite):
var dc = new LINQPad.User.TypedDataContext (IQToolkit.Data.DbEntityProvider.From
("IQToolkit.Data.Sqlite", #"Data Source=D:\SQLite.NET\nutshell.db",
"LINQPad.User.TypedDataContext"));
var customerCount = dc.Customers.Count();
This should get you started. Bear in mind the caveats, as stated in the answer to which you linked!