I am trying to learn Apache Calcite by following the RelBuilderExample with the storage layer being HSQLDB.
Unfortunately, I keep getting "Table Not Found exception" when i call builder.scan(tableName) API of Apache Calcite. When I query the data in HSQL directly using ResultSet rs = statement.executeQuery("SELECT * from file"); then i am able to retrieve the data. Here is the relevant code:
//I create an instance of RelBuilder using the Config defined below
RelBuilder builder = RelBuilder.create(config().build());
//This line throws me exception: org.apache.calcite.runtime.CalciteException: Table 'file' not found
builder = builder.scan("file");
/**
Building the configuration backed by HSQLDB
*/
public static Frameworks.ConfigBuilder config() throws Exception{
//Getting the ConnectionSpec for the in memory HSQLDB
//FileHSQLDB.URI = "jdbc:hsqldb:mem:intel"
//FileHSQLDB.USER = "user"
//FileHSQLDB.PASSWORD = "password"
final ConnectionSpec cs = new ConnectionSpec(FileHSQLDB.URI, FileHSQLDB.USER, FileHSQLDB.PASSWORD, "org.hsqldb.jdbcDriver", "intel");
//cs.url = "jdbc:hsqldb:mem:intel"
//cs.driver = "org.hsqldb.jdbcDriver"
//cs.username = "user"
//cs.password = "password"
DataSource dataSource = JdbcSchema.dataSource(cs.url, cs.driver, cs.username, cs.password);
Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement();
//This returns me 3 results
ResultSet rs = statement.executeQuery("SELECT * from file");
while(rs.next()) {
String id = rs.getString("file_id");
System.out.println(id);
}
// Next I create the rootSchema
SchemaPlus rootSchema = Frameworks.createRootSchema(true);
//I suspect that there is some issue in the below line. I think I
//am not using Apache Calcite APIs properly, but not sure what I
//am doing wrong.
rootSchema.add("intel", JdbcSchema.create(rootSchema, "intel", dataSource, cs.catalog, cs.schema));
return Frameworks.newConfigBuilder().defaultSchema(rootSchema);
Can someone please help me what I may be doing wrong.
If your table is file (lowercase) then make sure you quote the table name in the query, i.e. "SELECT * from \"file\"".
I have an sql server database table which has xml column name called "MESSAGE" and which will store xml data.
The database table look like,
Now I need to get this "MESSAGE" column data and save into System physical path as xml file(Ex: test.xml etc.,)
Any suggestion how to implement this using c#.net?
You could try something like this (using plain ADO.NET and a very basic SQL query):
static void Main(string[] args)
{
// get connection string from app./web.config
string connectionString = "server=.;database=yourDB;Integrated Security=SSPI;";
// define query
string query = "SELECT MESSAGE FROM dbo.SamTest WHERE ID = 1;";
// set up connection and command
using (SqlConnection conn = new SqlConnection(connectionString))
using (SqlCommand selectCmd = new SqlCommand(query, conn))
{
// open connection, execute query to get XML, close connection
conn.Open();
string xmlContents = selectCmd.ExecuteScalar().ToString();
conn.Close();
// define target file name
string targetFileName = #"C:\tmp\test.xml";
// write XML out to file
File.WriteAllText(targetFileName, xmlContents);
}
}
There is a problem. I have old database with some data, and by another side I have new database with new structure.
Now I need best way (ideas) how to copy data from one table to the another. Problem is some tables have max 1000 records some 32 000 some 640 000, and time to copy 5000+ is really long.
Any best practices ? Sample code below ...
public ActionResult ImportTable1()
{
var oldTable1 = context.OLDTABLE.ToList();
foreach (var item in oldTable1)
{
try
{
var cTable = contextNew.NEWTABLE.Where(p => p.fiel1 == item.field1).FirstOrDefault();
if (cTable == null)
{
NEWTABLE nTable = new NEWTABLE
{
field1 = item.field1,
field2 = item.field2
};
contextNew.NEWTABLE.Add(nTable);
}
else
{
cTable.field1 = item.field1
cTable.field2 = item.field2;
contextNew.Entry(cTable).State = EntityState.Modified;
}
IcontextNew.SaveChanges();
}
catch (DbEntityValidationException dbEx)
{
foreach (var validationErrors in dbEx.EntityValidationErrors)
{
foreach (var validationError in validationErrors.ValidationErrors)
{
_progresLog = ("Property: " + validationError.PropertyName + " Error: {1}" + validationError.ErrorMessage);
}
}
}
return PartialView();
}
... so bulk now
public void ExperimentalPartsBulk()
{
string msisDatabase = ConfigurationManager.ConnectionStrings["old"].ToString();
string newDatabase = ConfigurationManager.ConnectionStrings["new"].ToString();
SqlConnection sourceconnection = new SqlConnection(msisDatabase);
SqlConnection sourcedestination = new SqlConnection(newDatabase);
sourceconnection.Open();
SqlCommand cmd = new SqlCommand("Select * from ELEMENTS");
cmd.Connection = sourceconnection;
SqlDataReader reader = cmd.ExecuteReader();
//Connect to Destination DataBase
SqlConnection destinationConnection = new SqlConnection(newDatabase);
destinationConnection.Open();
SqlBulkCopy bulkCopy = new SqlBulkCopy(destinationConnection);
bulkCopy.DestinationTableName = "ELEMENTSNEW";
bulkCopy.ColumnMappings.Clear();
bulkCopy.ColumnMappings.Add("fielString1", "newString1");
bulkCopy.ColumnMappings.Add("fielString2", "newStrin2");
bulkCopy.ColumnMappings.Add("fielFloat1", "newINT1");
bulkCopy.WriteToServer(reader);
reader.Close();
sourceconnection.Close();
sourcedestination.Close();
}
problem now is w differences betwen two tables
fielString1 can be null, newString1 cant be |
fielFloat1 is float now is nullable but newINT1 not
How to import with some conditions or to the different types of field ?
Siwek,
Any loop as shown in the first code sample will failed due to performance issues.... as you pointed!
The right approach here is SQL approach. The idea is to "flush" all data to new DB. Flush mean that ALL records (5,000 or 500,000) are stored to new DB with one action! And avoid any loops during extracting, filtering, editing and saving of data, because 640,000 loops takes long time....
Bulk copy is one possible. Issue with bulk copy is that it's hard for you to filter and edit data in this object.
Use ADO.net DataSet to get data from old DB, filter it, edit it, and save it on memory and flush it to new DB. DataSet take one step per action (extracting, filtering, editing, etc. ! no loops).
Or, try SQL replication. Replication is the SQL mechanism to copy data from DB "A" table "oneTable" to another DB, "B" with a table "AnotherTable" with a different schema and rules. Try it. I can specify more if you think it's a reasonable solution for you. No code needed, it's can be created using wizard on SQL Management studio, and run whenever needed (via SQL Job agent).
You should seriously consider SSIS or bcp. Otherwise you are looking at a scenario whet you're pulling data from the source server all the way down to the client box where the .net code is executing, then pushing all off that data up to the destination server. Think of the bandwidth being consumed. If you can instead do an SSIS export into the destination, at least it would be eliminating an extra layer of concern.
If you absolutely must pull data down to the client, consider writing the data into bcp formatted files, and then bulkcopying them into the destination server.
I'm pretty sure that you'll find that both of these paths are significantly faster than using plain old ADO.NET sort of approaches.
Sorry for my English first of all. I have a problem and need help.
I have a simple tool made by myself on c#. This tool makes connect to local or remote firebird server (v.2.5). And my tool can create specified .fdb file (database) somewhere on the server.
Also I have a file with SQL statements (create table, triggers and so on). I want to execute this file after database was created. Executing this file will fill structure of user database - not data, only structure.
But then I try to execute my SQL script - firebird server returns a
SQL error code = -104 Token unknown line xxx column xxx.
That's the line on this CREATE TABLE SQL statement, for example:
CREATE TABLE tb1
(
col1 INTEGER NOT NULL,
col2 VARCHAR(36)
);
/* This next create statement causes an error */
CREATE TABLE tb2
(
col1 INTEGER NOT NULL,
col2 VARCHAR(36)
);
If I will leave only one create statement in my file - all will be good... I don't know how I explained (it's clear or not)) - another words - why can't I execute full query with many create statements in one transaction? There is my main method which executes query:
public static string Do(string conString, string query)
{
using (FbConnection conn = new FbConnection())
{
try
{
conn.ConnectionString = conString;
conn.Open();
FbTransaction trans = conn.BeginTransaction();
FbCommand cmd = new FbCommand(query, conn, trans);
cmd.ExecuteNonQuery();
trans.Commit();
}
catch (Exception ex)
{
System.Windows.MessageBox.Show(ex.ToString());
return "Transaction Fail";
}
}
return "Transaction Commited";
}
There is a query is my SQL file.
As Victor already stated in his final comment, you can use the FBScript class for batch execution.
I was just confronted with the same task. This question pointed me in the right direction but i had to do some further digging.
I this example, the source of the statements is a external script file:
private void ExecuteScript(FbConnection myConnection, string scriptPath) {
if (!File.Exists(scriptPath))
throw new FileNotFoundException("Script not found", scriptPath);
FileInfo file = new FileInfo(scriptPath);
string script = file.OpenText().ReadToEnd();
// use FbScript to parse all statements
FbScript fbs = new FbScript(script);
fbs.Parse();
// execute all statements
FbBatchExecution fbe = new FbBatchExecution(myConnection, fbs);
fbe.Execute(true);
}
This will work fine, but you may wonder why this whole thing isn't surrounded by a transaction. Actually there is no support to "bind" FbBatchExecution to a transaction directly.
The first thing i tried was this (will not work)
private void ExecuteScript(FbConnection myConnection, string scriptPath) {
using (FbTransaction myTransaction = myConnection.BeginTransaction()) {
if (!File.Exists(scriptPath))
throw new FileNotFoundException("Script not found", scriptPath);
FileInfo file = new FileInfo(scriptPath);
string script = file.OpenText().ReadToEnd();
// use FbScript to parse all statements
FbScript fbs = new FbScript(script);
fbs.Parse();
// execute all statements
FbBatchExecution fbe = new FbBatchExecution(myConnection, fbs);
fbe.Execute(true);
myTransaction.Commit();
}
}
This will result in an exception stating: "Execute requires the Command object to have a Transaction object when the Connection object assigned to the command is in a pending local transaction. The Transaction property of the Command has not been initialized."
This means nothing more than that the commands that are executed by FbBatchExecution are not assigned to our local transaction that is surrounding the code block. What helps here is that that FbBatchExecution provides
the event CommandExecuting where we can intercept every command and assign our local transaction like this:
private void ExecuteScript(FbConnection myConnection, string scriptPath) {
using (FbTransaction myTransaction = myConnection.BeginTransaction()) {
if (!File.Exists(scriptPath))
throw new FileNotFoundException("Script not found", scriptPath);
FileInfo file = new FileInfo(scriptPath);
string script = file.OpenText().ReadToEnd();
// use FbScript to parse all statements
FbScript fbs = new FbScript(script);
fbs.Parse();
// execute all statements
FbBatchExecution fbe = new FbBatchExecution(myConnection, fbs);
fbe.CommandExecuting += delegate(object sender, CommandExecutingEventArgs args) {
args.SqlCommand.Transaction = myTransaction;
};
fbe.Execute(true);
// myTransaction.Commit();
}
}
Note that i have uncommented the myTransaction.Commit() line. I was a little bit surprised by this behavior, but if you keep that line the transaction will throw an exception stating that it has already been committed. The bool parameter fbe.Execute(true) is named "autoCommit", but changing this to false seems to have no effect.
I would like some feedback if you see any potential issues with assigning the local transaction this way, or if it has any benefits at all or could as well be omitted.
Probably error in launching two create statements in one batch. Would it work if you break it to separate queries? Does it work in your SQL tool?
I am trying to search an XML field within a table, This is not supported with EF.
Without using pure Ado.net is possible to have native SQL support with EF?
For .NET Framework version 4 and above: use ObjectContext.ExecuteStoreCommand() if your query returns no results, and use ObjectContext.ExecuteStoreQuery if your query returns results.
For previous .NET Framework versions, here's a sample illustrating what to do. Replace ExecuteNonQuery() as needed if your query returns results.
static void ExecuteSql(ObjectContext c, string sql)
{
var entityConnection = (System.Data.EntityClient.EntityConnection)c.Connection;
DbConnection conn = entityConnection.StoreConnection;
ConnectionState initialState = conn.State;
try
{
if (initialState != ConnectionState.Open)
conn.Open(); // open connection if not already open
using (DbCommand cmd = conn.CreateCommand())
{
cmd.CommandText = sql;
cmd.ExecuteNonQuery();
}
}
finally
{
if (initialState != ConnectionState.Open)
conn.Close(); // only close connection if not initially open
}
}
Using Entity Framework 5.0 you can use ExecuteSqlCommand to execute multi-line/multi-command pure SQL statements. This way you won't need to provide any backing object to store the returned value since the method returns an int (the result returned by the database after executing the command).
Sample:
context.Database.ExecuteSqlCommand(#
"-- Script Date: 10/1/2012 3:34 PM - Generated by ExportSqlCe version 3.5.2.18
SET IDENTITY_INSERT [Students] ON;
INSERT INTO [Students] ([StudentId],[FirstName],[LastName],[BirthDate],[Address],[Neighborhood],[City],[State],[Phone],[MobilePhone],[Email],[Enrollment],[Gender],[Status]) VALUES (12,N'First Name',N'SecondName',{ts '1988-03-02 00:00:00.000'},N'RUA 19 A, 60',N'MORADA DO VALE',N'BARRA DO PIRAÍ',N'Rio de Janeiro',N'3346-7125',NULL,NULL,{ts '2011-06-04 21:25:26.000'},2,1);
INSERT INTO [Students] ([StudentId],[FirstName],[LastName],[BirthDate],[Address],[Neighborhood],[City],[State],[Phone],[MobilePhone],[Email],[Enrollment],[Gender],[Status]) VALUES (13,N'FirstName',N'LastName',{ts '1976-04-12 00:00:00.000'},N'RUA 201, 2231',N'RECANTO FELIZ',N'BARRA DO PIRAÍ',N'Rio de Janeiro',N'3341-6892',NULL,NULL,{ts '2011-06-04 21:38:38.000'},2,1);
");
For more on this, take a look here: Entity Framework Code First: Executing SQL files on database creation
For Entity Framework 5 use context.Database.SqlQuery.
And for Entity Framework 4 use context.ExecuteStoreQuery
the following code:
public string BuyerSequenceNumberMax(int buyerId)
{
string sequenceMaxQuery = "SELECT TOP(1) btitosal.BuyerSequenceNumber FROM BuyerTakenItemToSale btitosal " +
"WHERE btitosal.BuyerID = " + buyerId +
"ORDER BY CONVERT(INT,SUBSTRING(btitosal.BuyerSequenceNumber,7, LEN(btitosal.BuyerSequenceNumber))) DESC";
var sequenceQueryResult = context.Database.SqlQuery<string>(sequenceMaxQuery).FirstOrDefault();
string buyerSequenceNumber = string.Empty;
if (sequenceQueryResult != null)
{
buyerSequenceNumber = sequenceQueryResult.ToString();
}
return buyerSequenceNumber;
}
To return a List use the following code:
public List<PanelSerialList> PanelSerialByLocationAndStock(string locationCode, byte storeLocation, string itemCategory, string itemCapacity, byte agreementType, string packageCode)
{
string panelSerialByLocationAndStockQuery = "SELECT isws.ItemSerialNo, im.ItemModel " +
"FROM Inv_ItemMaster im " +
"INNER JOIN " +
"Inv_ItemStockWithSerialNoByLocation isws " +
" ON im.ItemCode = isws.ItemCode " +
" WHERE isws.LocationCode = '" + locationCode + "' AND " +
" isws.StoreLocation = " + storeLocation + " AND " +
" isws.IsAvailableInStore = 1 AND " +
" im.ItemCapacity = '" + itemCapacity + "' AND " +
" isws.ItemSerialNo NOT IN ( " +
" Select sp.PanelSerialNo From Special_SpecialPackagePriceForResale sp " +
" Where sp.PackageCode = '" + packageCode + "' )";
return context.Database.SqlQuery<PanelSerialList>(panelSerialByLocationAndStockQuery).ToList();
}
Keep it simple
using (var context = new MyDBEntities())
{
var m = context.ExecuteStoreQuery<MyDataObject>("Select * from Person", string.Empty);
//Do anything you wonna do with
MessageBox.Show(m.Count().ToString());
}
public class RaptorRepository<T>
where T : class
{
public RaptorRepository()
: this(new RaptorCoreEntities())
{
}
public RaptorRepository(ObjectContext repositoryContext)
{
_repositoryContext = repositoryContext ?? new RaptorCoreEntities();
_objectSet = repositoryContext.CreateObjectSet<T>();
}
private ObjectContext _repositoryContext;
private ObjectSet<T> _objectSet;
public ObjectSet<T> ObjectSet
{
get
{
return _objectSet;
}
}
public void DeleteAll()
{
_repositoryContext
.ExecuteStoreCommand("DELETE " + _objectSet.EntitySet.ElementType.Name);
}
}
So what do we say about all this in 2017? 80k consultations suggests that running a SQL request in EF is something a lot of folk want to do. But why? For what benefit?
Justin, a guru with 20 times my reputation, in the accepted answer gives us a static method that looks line for line like the equivalent ADO code. Be sure to copy it well because there are a few subtleties to not get wrong. And you're obliged to concatenate your query with your runtime parameters since there's no provision for proper parameters. So all users of this method will be constructing their SQL with string methods (fragile, untestable, sql injection), and none of them will be unit testing.
The other answers have the same faults, only moreso. SQL buried in double quotes. SQL injection opportunities liberally scattered around. Esteemed peers, this is absolutely savage behaviour. If this was C# being generated, there would be a flame war. We don't even accept generating HTML this way, but somehow its OK for SQL. I know that query parameters were not the subject of the question, but we copy and reuse what we see, and the answers here are both models and testaments to what folk are doing.
Has EF melted our brains? EF doesn't want you to use SQL, so why use EF to do SQL.
Wanting to use SQL to talk to a relational DB is a healthy, normal impulse in adults. QueryFirst shows how this could be done intelligently, your sql in .sql file, validated as you type, with intellisense for tables and columns. The C# wrapper is generated by the tool, so your queries become discoverable in code, with intellisense for your inputs and results. End to end strong typing, without ever having to worry about a type. No need to ever remember a column name, or its index. And there are numerous other benefits... The temptation to concatenate is removed. The possibility of mishandling your connections also. All your queries and the code that accesses them are continuously integration-tested against your dev DB. Schema changes in your DB pop up as compile errors in your app. We even generate a self test method in the wrapper, so you can test new versions of your app against existing production databases, rather than waiting for the phone to ring. Anyone still need convincing?
Disclaimer: I wrote QueryFirst :-)