I have an SQL script file which drops and recreates various tables as well as inserts various records into these tables. The script runs fine when executing in the SQL query console however I need it to be executed by the Entity Manager.
Any idea's on how I would be able to do this?
Thanks,
H
Late to the party but here's how I do it. Couple of things to note here:
My SQL file ("sql-queries.sql") is on the classpath - you could do this any other way that will get you an input stream...
My SQL file has 1 statement per line
I'm manually beginning/committing transactions, one for each line/statement in the file
Here's the method to execute the file:
void executeStatements(BufferedReader br, EntityManager entityManager) throws IOException {
String line;
while( (line = br.readLine()) != null )
{
entityManager.getTransaction().begin();
entityManager.createNativeQuery(line).executeUpdate();
entityManager.getTransaction().commit();
}
}
Here's how I call it:
InputStream sqlFileInputStream = Thread.currentThread().getContextClassLoader()
.getResourceAsStream("geo-data.sql");
// Convert input stream to something that can be read line-by-line
BufferedReader sqlFileBufferedReader = new BufferedReader( new InputStreamReader(sqlFileInputStream));
executeStatements(sqlFileBufferedReader, dao.getEntityManager());
I tested nominally with 1 transaction instead of 1 per statement (note that this means that 1 bad query will break everything) and the time to execute is the same:
void executeStatements(BufferedReader br, EntityManager entityManager) throws IOException {
String line;
entityManager.getTransaction().begin();
while( (line = br.readLine()) != null )
{
entityManager.createNativeQuery(line).executeUpdate();
}
entityManager.getTransaction().commit();
}
Related
This problem is frustrating me so much, that I've finally created a Stackoverflow Account - this is my first question here, go easy on me.
I've got an SQL Script like this:
select * from d_table
where id = :line;
I'd like to execute that query from a groovy script. So far I've written this:
static void main(String[] args) {
String dbURL = 'some.db:Port:serviceName'
String dbUser = 'dbUser'
String dbPw = 'aReallyStrongPassword'
// connect to SQL Instance and DB
def sql = Sql.newInstance(dbURL, dbUser, dbPw, 'oracle.jdbc.OracleDriver')
// read Inputfile from local dir
new File("inputfile.csv").eachLine { inputLine ->
// loop over Inputfile
try {
// read SQL Script from local dir
String sqlString = new File("script.sql").getText("UTF-8")
// execute SQL Script, replace :line with input-ID
resp = sql.execute sqlString, line: inputLine
} catch(Exception e) {
println(e)
}
}
// close SQL Session
sql.close()
}
But I'll always get an error like this:
WARNING: Failed to execute: select * from d_table
where id = :line; because: ORA-00933: SQL command not properly ended
I've got the feeling the query is not built properly, but I don't know whats the problem?
Using groovy 3.0.4.
Here the Main class :
public class Main {
// JDBC driver name and database URL
static final String JDBC_DRIVER = "org.h2.Driver";
static final String DB_URL = "jdbc:h2:mem:default";
// Database credentials
static final String USER = "sa";
static final String PASS = "";
public static void main(String[] args) {
Connection conn = null;
Statement stmt = null;
try {
// STEP 1: Register JDBC driver
Class.forName(JDBC_DRIVER);
//STEP 2: Open a connection
System.out.println("Connecting to database...");
conn = DriverManager.getConnection(DB_URL, USER, PASS);
//STEP 3: Execute a query
ScriptRunner sr = new ScriptRunner(conn);
//Creating a reader object
Reader reader = new BufferedReader(new FileReader("C:\\PROJECTS\\src\\main\\resources\\createDatabaseAndUser.sql"));
//Running the script
sr.runScript(reader);
// STEP 4: Clean-up environment
conn.close();
} catch (SQLException se) {
//Handle errors for JDBC
se.printStackTrace();
} catch (Exception e) {
//Handle errors for Class.forName
e.printStackTrace();
} finally {
//finally block used to close resources
try {
if (stmt != null) stmt.close();
} catch (SQLException se2) {
} // nothing we can do
try {
if (conn != null) conn.close();
} catch (SQLException se) {
se.printStackTrace();
} //end finally try
} //end try
System.out.println("Goodbye!");
}
}
createDatabaseAndUser.sql file which located in resourses folder :
CREATE SCHEMA database;
CREATE USER ADMIN PASSWORD 'abc';
I am trying to run my SQL script but receive and ERROR.
Here all output :
Connecting to database...
CREATE SCHEMA database
Error executing: CREATE SCHEMA database.Cause: org.h2.jdbc.JdbcSQLSyntaxErrorException: Syntax error
in SQL statement "CREATE[*] SCHEMA DATABASE"; SQL statement:
CREATE USER ADMIN PASSWORD 'abc'
CREATE SCHEMA database
[42000-199]
Goodbye!
My SQL Dialect is H2 and I use embedded H2 database in Intelij IDEA.
I just begin learning SQL and SQL scripts, so sorry in advance if question is stupid.
It looks like the output from the tool that you use is garbled somehow. There is CREATE[*] SCHEMA DATABASE command ([*] is a marker from H2), but the source SQL is listed as
CREATE USER ADMIN PASSWORD 'abc'
CREATE SCHEMA database
and it is obliviously invalid due to missing semicolon between two commands.
Both places should be the same (with exception for [*] mark) and the error message should be
Syntax error in SQL statement "CREATE USER ADMIN PASSWORD 'abc'
CREATE[*] SCHEMA DATABASE "; SQL statement:
CREATE USER ADMIN PASSWORD 'abc'
CREATE SCHEMA database [42000-199]
So you ether have a missing semicolon in your source file, or this tool removes it. Actually you don't need to use the third-party tool, you can execute the script directly in H2:
Statement st = conn.createStatement();
st.execute("RUNSCRIPT FROM 'C:\\PROJECTS\\src\\main\\resources\\createDatabaseAndUser.sql'");
I have custom extractor, and I'm trying to log some messages from it.
I've tried obvious things like Console.WriteLine, but cannot find where output is. However, I found some system logs in adl://<my_DLS>.azuredatalakestore.net/system/jobservice/jobs/Usql/.../<my_job_id>/.
How can I log something? Is it possible to specify log file somewhere on Data Lake Store or Blob Storage Account?
A recent release of U-SQL has added diagnostic logging for UDOs. See the release notes here.
// Enable the diagnostics preview feature
SET ##FeaturePreviews = "DIAGNOSTICS:ON";
// Extract as one column
#input =
EXTRACT col string
FROM "/input/input42.txt"
USING new Utilities.MyExtractor();
#output =
SELECT *
FROM #input;
// Output the file
OUTPUT #output
TO "/output/output.txt"
USING Outputters.Tsv(quoting : false);
This was my diagnostic line from the UDO:
Microsoft.Analytics.Diagnostics.DiagnosticStream.WriteLine(System.String.Format("Concatenations done: {0}", i));
This is the whole UDO:
using System.Collections.Generic;
using System.IO;
using System.Text;
using Microsoft.Analytics.Interfaces;
namespace Utilities
{
[SqlUserDefinedExtractor(AtomicFileProcessing = true)]
public class MyExtractor : IExtractor
{
//Contains the row
private readonly Encoding _encoding;
private readonly byte[] _row_delim;
private readonly char _col_delim;
public MyExtractor()
{
_encoding = Encoding.UTF8;
_row_delim = _encoding.GetBytes("\n\n");
_col_delim = '|';
}
public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output)
{
string s = string.Empty;
string x = string.Empty;
int i = 0;
foreach (var current in input.Split(_row_delim))
{
using (System.IO.StreamReader streamReader = new StreamReader(current, this._encoding))
{
while ((s = streamReader.ReadLine()) != null)
{
//Strip any line feeds
//s = s.Replace("/n", "");
// Concatenate the lines
x += s;
i += 1;
}
Microsoft.Analytics.Diagnostics.DiagnosticStream.WriteLine(System.String.Format("Concatenations done: {0}", i));
//Create the output
output.Set<string>(0, x);
yield return output.AsReadOnly();
// Reset
x = string.Empty;
}
}
}
}
}
And these were my results found in the following directory:
/system/jobservice/jobs/Usql/2017/10/20.../diagnosticstreams
good question. I have been asking myself the same thing. This is theoretical, but I think it would work (I'll updated if I find differently).
One very hacky way is that you could insert rows into a table with your log messages as a string column. Then you can select those out and filter based on some log_producer_id column. You also get the benefit of logging if part of the script works, but later parts do not assuming the failure does not roll back. Table can be dumped at end as well to file.
For the error cases, you can use the Job Manager in ADLA to open the job graph and then view the job output. The errors often have detailed information for data-related errors (e.g. row number in file with error and a octal/hex/ascii dump of the row with issue marked with ###).
Hope this helps,
J
ps. This isn't a comment or an answer really, since I don't have working code. Please provide feedback if the above ideas are wrong.
I have a very strange error with dapper:
there is already an open DataReader associated with this Command
which must be closed first
But I don't use DataReader! I just call select query on my server application and take first result:
//How I run query:
public static T SelectVersion(IDbTransaction transaction = null)
{
return DbHelper.DataBase.Connection.Query<T>("SELECT * FROM [VersionLog] WHERE [Version] = (SELECT MAX([Version]) FROM [VersionLog])", null, transaction, commandTimeout: DbHelper.CommandTimeout).FirstOrDefault();
}
//And how I call this method:
public Response Upload(CommitRequest message) //It is calling on server from client
{
//Prepearing data from CommitRequest
using (var tr = DbHelper.DataBase.Connection.BeginTransaction(IsolationLevel.Serializable))
{
int v = SelectQueries<VersionLog>.SelectVersion(tr) != null ? SelectQueries<VersionLog>.SelectVersion(tr).Version : 0; //Call my query here
int newVersion = v + 1; //update version
//Saving changes from CommitRequest to db
//Updated version saving to base too, maybe it is problem?
return new Response
{
Message = String.Empty,
ServerBaseVersion = versionLog.Version,
};
}
}
}
And most sadly that this exception appearing in random time, I think what problem in concurrent access to server from two clients.
Please help.
This some times happens if the model and database schema are not matching and an exception is being raised inside Dapper.
If you really want to get into this, best way is to include dapper source in your project and debug.
Sorry for my English first of all. I have a problem and need help.
I have a simple tool made by myself on c#. This tool makes connect to local or remote firebird server (v.2.5). And my tool can create specified .fdb file (database) somewhere on the server.
Also I have a file with SQL statements (create table, triggers and so on). I want to execute this file after database was created. Executing this file will fill structure of user database - not data, only structure.
But then I try to execute my SQL script - firebird server returns a
SQL error code = -104 Token unknown line xxx column xxx.
That's the line on this CREATE TABLE SQL statement, for example:
CREATE TABLE tb1
(
col1 INTEGER NOT NULL,
col2 VARCHAR(36)
);
/* This next create statement causes an error */
CREATE TABLE tb2
(
col1 INTEGER NOT NULL,
col2 VARCHAR(36)
);
If I will leave only one create statement in my file - all will be good... I don't know how I explained (it's clear or not)) - another words - why can't I execute full query with many create statements in one transaction? There is my main method which executes query:
public static string Do(string conString, string query)
{
using (FbConnection conn = new FbConnection())
{
try
{
conn.ConnectionString = conString;
conn.Open();
FbTransaction trans = conn.BeginTransaction();
FbCommand cmd = new FbCommand(query, conn, trans);
cmd.ExecuteNonQuery();
trans.Commit();
}
catch (Exception ex)
{
System.Windows.MessageBox.Show(ex.ToString());
return "Transaction Fail";
}
}
return "Transaction Commited";
}
There is a query is my SQL file.
As Victor already stated in his final comment, you can use the FBScript class for batch execution.
I was just confronted with the same task. This question pointed me in the right direction but i had to do some further digging.
I this example, the source of the statements is a external script file:
private void ExecuteScript(FbConnection myConnection, string scriptPath) {
if (!File.Exists(scriptPath))
throw new FileNotFoundException("Script not found", scriptPath);
FileInfo file = new FileInfo(scriptPath);
string script = file.OpenText().ReadToEnd();
// use FbScript to parse all statements
FbScript fbs = new FbScript(script);
fbs.Parse();
// execute all statements
FbBatchExecution fbe = new FbBatchExecution(myConnection, fbs);
fbe.Execute(true);
}
This will work fine, but you may wonder why this whole thing isn't surrounded by a transaction. Actually there is no support to "bind" FbBatchExecution to a transaction directly.
The first thing i tried was this (will not work)
private void ExecuteScript(FbConnection myConnection, string scriptPath) {
using (FbTransaction myTransaction = myConnection.BeginTransaction()) {
if (!File.Exists(scriptPath))
throw new FileNotFoundException("Script not found", scriptPath);
FileInfo file = new FileInfo(scriptPath);
string script = file.OpenText().ReadToEnd();
// use FbScript to parse all statements
FbScript fbs = new FbScript(script);
fbs.Parse();
// execute all statements
FbBatchExecution fbe = new FbBatchExecution(myConnection, fbs);
fbe.Execute(true);
myTransaction.Commit();
}
}
This will result in an exception stating: "Execute requires the Command object to have a Transaction object when the Connection object assigned to the command is in a pending local transaction. The Transaction property of the Command has not been initialized."
This means nothing more than that the commands that are executed by FbBatchExecution are not assigned to our local transaction that is surrounding the code block. What helps here is that that FbBatchExecution provides
the event CommandExecuting where we can intercept every command and assign our local transaction like this:
private void ExecuteScript(FbConnection myConnection, string scriptPath) {
using (FbTransaction myTransaction = myConnection.BeginTransaction()) {
if (!File.Exists(scriptPath))
throw new FileNotFoundException("Script not found", scriptPath);
FileInfo file = new FileInfo(scriptPath);
string script = file.OpenText().ReadToEnd();
// use FbScript to parse all statements
FbScript fbs = new FbScript(script);
fbs.Parse();
// execute all statements
FbBatchExecution fbe = new FbBatchExecution(myConnection, fbs);
fbe.CommandExecuting += delegate(object sender, CommandExecutingEventArgs args) {
args.SqlCommand.Transaction = myTransaction;
};
fbe.Execute(true);
// myTransaction.Commit();
}
}
Note that i have uncommented the myTransaction.Commit() line. I was a little bit surprised by this behavior, but if you keep that line the transaction will throw an exception stating that it has already been committed. The bool parameter fbe.Execute(true) is named "autoCommit", but changing this to false seems to have no effect.
I would like some feedback if you see any potential issues with assigning the local transaction this way, or if it has any benefits at all or could as well be omitted.
Probably error in launching two create statements in one batch. Would it work if you break it to separate queries? Does it work in your SQL tool?