How can I do this all in one query? Nhibernate - nhibernate

I have a list of Ids and I want to get all the rows back in one query. As a list of objects(So a List of Products or whatever).
I tried
public List<TableA> MyMethod(List<string> keys)
{
var query = "SELECT * FROM TableA WHERE Keys IN (:keys)";
var a = session.CreateQuery(query).SetParameter("keys", keys).List();
return a; // a is a IList but not of TableA. So what do I do now?
}
but I can't figure out how to return it as a list of objects. Is this the right way?

List<TableA> result = session.CreateQuery(query)
.SetParameterList("keys", keys)
.List<TableA>();

Howeever there could be a limitation in this query if number of ":keys" exceed more than 1000 (incase of oracle not sure with other dbs) so i would recommend to use ICriteria instead of CreateQuery- native sqls.
Do something like this,
[TestFixture]
public class ThousandIdsNHibernateQuery
{
[Test]
public void TestThousandIdsNHibernateQuery()
{
//Keys contains 1000 ids not included here.
var keys = new List<decimal>();
using (ISession session = new Session())
{
var tableCirt = session.CreateCriteria(typeof(TableA));
if (keys.Count > 1000)
{
var listsList = new List<List<decimal>>();
//Get first 1000.
var first1000List = keys.GetRange(0, 1000);
//Split next keys into 1000 chuncks.
for (int i = 1000; i < keys.Count; i++)
{
if ((i + 1)%1000 == 0)
{
var newList = new List<decimal>();
newList.AddRange(keys.GetRange(i - 999, 1000));
listsList.Add(newList);
}
}
ICriterion firstExp = Expression.In("Key", first1000List);
ICriterion postExp = null;
foreach (var list in listsList)
{
postExp = Expression.In("Key", list);
tableCirt.Add(Expression.Or(firstExp, postExp));
firstExp = postExp;
}
tableCirt.Add(postExp);
}
else
{
tableCirt.Add(Expression.In("key", keys));
}
var results = tableCirt.List<TableA>();
}
}
}

Related

Is it possible to return an array of dictionaries?

I wrote a function that will return a single sql record as a dictionary. Is it possible to return an array of dictionaries so I can return multiple records in this way?
public static async Task<Dictionary<string, string>> SQLMultiRecordToDictionary(string TableName, string SearchField, string SearchValue)
{
Dictionary<string, string> QueryResult = new Dictionary<string, string>();
// is TableName sane
if (!IsTextSane(TableName)) { return QueryResult; }
//
await using (var connection = new SqliteConnection("Data Source=" + dbFullPathName))
{
connection.Open();
SqliteCommand sqlcmd = connection.CreateCommand();
sqlcmd.CommandText = "SELECT * FROM " + TableName + " WHERE " + SearchField + "=#SearchValue";
sqlcmd.Parameters.AddWithValue("#SearchValue", SearchValue);
SqliteDataReader sqlreader = sqlcmd.ExecuteReader();
// generate dictionary keys with blank values
// this prevents key not existing issues when no record is returned
// i prefer no/blank values in the keys when no record returned for this project
for (int i = 0; i < sqlreader.FieldCount; i++)
{
QueryResult.Add(sqlreader.GetName(i), ""); // blank value
}
// add the values to the keys
while (sqlreader.Read())
{
for (int i = 0; i <= sqlreader.FieldCount - 1; i++)
{
QueryResult[sqlreader.GetName(i)] = sqlreader.GetString(i);
}
}
return QueryResult;
}
}
The working end result thanks to Tisa:
public static async Task<List<Dictionary<string, string>>> SQLMultiRecordToDictionaryList(string TableName, string SearchField, string SearchValue)
{
List<Dictionary<string, string>> QueryResult = new List<Dictionary<string, string>>();
Dictionary<string, string> SQLRecord = new Dictionary<string, string>();
//
// is TableName sane, if not return nothing
if (!IsTextSane(TableName)) { return QueryResult; }
//
await using (var connection = new SqliteConnection("Data Source=" + dbFullPathName))
{
connection.Open();
SqliteCommand sqlcmd = connection.CreateCommand();
sqlcmd.CommandText = "SELECT * FROM " + TableName + " WHERE " + SearchField + "=#SearchValue";
sqlcmd.Parameters.AddWithValue("#SearchValue", SearchValue);
SqliteDataReader sqlreader = sqlcmd.ExecuteReader();
// generate dictionary keys with blank values if no rows
// this prevents key not existing issues when no record is returned
// i prefer no/blank values in the keys when no record returned for this project
if (!sqlreader.HasRows)
{
for (int i = 0; i < sqlreader.FieldCount; i++)
{
SQLRecord.Add(sqlreader.GetName(i), ""); // blank value
}
QueryResult.Add(SQLRecord);
}
//
// add the values to the keys if there are rows (this doesn't run if no rows returned)
while (sqlreader.Read())
{
SQLRecord = new Dictionary<string, string>();
for (int i = 0; i <= sqlreader.FieldCount - 1; i++)
{
SQLRecord.Add(sqlreader.GetName(i), sqlreader.GetString(i));
}
QueryResult.Add(SQLRecord);
}
return QueryResult;
}
}

Google BigQuery returns only partial table data with C# application using .net Client Library

I am trying to execute the query (Basic select statement with 10 fields). My table contains more than 500k rows. C# application returns the response with only 4260 rows. However Web UI returns all the records.
Why my code returns only partial data, What is the best way to select all the records and load into C# Data Table? If there is any code snippet it would be more helpful to me.
using Google.Apis.Auth.OAuth2;
using System.IO;
using System.Threading;
using Google.Apis.Bigquery.v2;
using Google.Apis.Bigquery.v2.Data;
using System.Data;
using Google.Apis.Services;
using System;
using System.Security.Cryptography.X509Certificates;
namespace GoogleBigQuery
{
public class Class1
{
private static void Main()
{
try
{
Console.WriteLine("Start Time: {0}", DateTime.Now.ToString());
String serviceAccountEmail = "SERVICE ACCOUNT EMAIL";
var certificate = new X509Certificate2(#"KeyFile.p12", "notasecret", X509KeyStorageFlags.Exportable);
ServiceAccountCredential credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(serviceAccountEmail)
{
Scopes = new[] { BigqueryService.Scope.Bigquery, BigqueryService.Scope.BigqueryInsertdata, BigqueryService.Scope.CloudPlatform, BigqueryService.Scope.DevstorageFullControl }
}.FromCertificate(certificate));
BigqueryService Service = new BigqueryService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "PROJECT NAME"
});
string query = "SELECT * FROM [publicdata:samples.shakespeare]";
JobsResource j = Service.Jobs;
QueryRequest qr = new QueryRequest();
string ProjectID = "PROJECT ID";
qr.Query = query;
qr.MaxResults = Int32.MaxValue;
qr.TimeoutMs = Int32.MaxValue;
DataTable DT = new DataTable();
int i = 0;
QueryResponse response = j.Query(qr, ProjectID).Execute();
string pageToken = null;
if (response.JobComplete == true)
{
if (response != null)
{
int colCount = response.Schema.Fields.Count;
if (DT == null)
DT = new DataTable();
if (DT.Columns.Count == 0)
{
foreach (var Column in response.Schema.Fields)
{
DT.Columns.Add(Column.Name);
}
}
pageToken = response.PageToken;
if (response.Rows != null)
{
foreach (TableRow row in response.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
while (true)
{
int StartIndexForQuery = DT.Rows.Count;
Google.Apis.Bigquery.v2.JobsResource.GetQueryResultsRequest SubQR = Service.Jobs.GetQueryResults(response.JobReference.ProjectId, response.JobReference.JobId);
SubQR.StartIndex = (ulong)StartIndexForQuery;
//SubQR.MaxResults = Int32.MaxValue;
GetQueryResultsResponse QueryResultResponse = SubQR.Execute();
if (QueryResultResponse != null)
{
if (QueryResultResponse.Rows != null)
{
foreach (TableRow row in QueryResultResponse.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
if (null == QueryResultResponse.PageToken)
{
break;
}
}
else
{
break;
}
}
}
else
{
Console.WriteLine("Response is null");
}
}
int TotalCount = 0;
if (DT != null && DT.Rows.Count > 0)
{
TotalCount = DT.Rows.Count;
}
else
{
TotalCount = 0;
}
Console.WriteLine("End Time: {0}", DateTime.Now.ToString());
Console.WriteLine("No. of records readed from google bigquery service: " + TotalCount.ToString());
}
catch (Exception e)
{
Console.WriteLine("Error Occurred: " + e.Message);
}
Console.ReadLine();
}
}
}
In this Sample Query get the results from public data set, In table contains 164656 rows but response returns 85000 rows only for the first time, then query again to get the second set of results. (But not known this is the only solution to get all the results).
In this sample contains only 4 fields, even-though it does not return all rows, in my case table contains more than 15 fields, I get response of ~4000 rows out of ~10k rows, I need to query again and again to get the remaining results for selecting 1000 rows takes time up to 2 minutes in my methodology so I am expecting best way to select all the records within single response.
Answer from User #:Pentium10
There is no way to run a query and select a large response in a single shot. You can either paginate the results, or if you can create a job to export to files, then use the files generated in your app. Exporting is free.
Step to run a large query and export results to files stored on GCS:
1) Set allowLargeResults to true in your job configuration. You must also specify a destination table with the allowLargeResults flag.
Example:
"configuration":
{
"query":
{
"allowLargeResults": true,
"query": "select uid from [project:dataset.table]"
"destinationTable": [project:dataset.table]
}
}
2) Now your data is in a destination table you set. You need to create a new job, and set the export property to be able to export the table to file(s). Exporting is free, but you need to have Google Cloud Storage activated to put the resulting files there.
3) In the end you download your large files from GCS.
It my turn to design the solution for better results.
Hoping this might help someone. One could retrieve next set of paginated result using PageToken. Here is the sample code for how to use PageToken. Although, I liked the idea of exporting for free. Here, I write rows to flat file but you could add them to your DataTable. Obviously, it is a bad idea to keep large DataTable in memory though.
public void ExecuteSQL(BigqueryService bqservice, String ProjectID)
{
string sSql = "SELECT r.Dealname, r.poolnumber, r.loanid FROM [MBS_Dataset.tblRemitData] R left join each [MBS_Dataset.tblOrigData] o on R.Dealname = o.Dealname and R.Poolnumber = o.Poolnumber and R.LoanID = o.LoanID Order by o.Dealname, o.poolnumber, o.loanid limit 100000";
QueryRequest _r = new QueryRequest();
_r.Query = sSql;
QueryResponse _qr = bqservice.Jobs.Query(_r, ProjectID).Execute();
string pageToken = null;
if (_qr.JobComplete != true)
{
//job not finished yet! expecting more data
while (true)
{
var resultReq = bqservice.Jobs.GetQueryResults(_qr.JobReference.ProjectId, _qr.JobReference.JobId);
resultReq.PageToken = pageToken;
var result = resultReq.Execute();
if (result.JobComplete == true)
{
WriteRows(result.Rows, result.Schema.Fields);
pageToken = result.PageToken;
if (pageToken == null)
break;
}
}
}
else
{
List<string> _fieldNames = _qr.Schema.Fields.ToList().Select(x => x.Name).ToList();
WriteRows(_qr.Rows, _qr.Schema.Fields);
}
}
The Web UI automatically flattens the data. This means that you see multiple rows for each nested field.
When you run the same query via the API, it won't be flattened, and you get fewer rows, as the nested fields are returned as objects. You should check if this is the case at you.
The other is that indeed you need to paginate through the results. Paging through list results has this explained.
If you want to do only one job, than you should write your query ouput to a table, than export the table as JSON, and download the export from GCS.

Topic views do not show up in jTable

I try to code a forum using java swing. Firstly, on click for the jTable, it will lead to eForumContent.class which pass in the id together.
jTable.addMouseListener(new java.awt.event.MouseAdapter() {
public void mouseClicked(java.awt.event.MouseEvent e) {
int id = 0;
eForumTopics topics = new eForumTopics(id);
topics.retrieveDiscussion();
eForumThreadContent myWindow = new eForumThreadContent(id);
myWindow.getJFrame().setVisible(true);
}
});
I initialize the id first. But I set my id in database table to auto number. Then I call the retrieveDiscussion method to get the id from database and pass it to eForumThreadContent. Here are my codes for retrieveDiscussion method.
public boolean retrieveDiscussion(){
boolean success = false;
ResultSet rs = null;
DBController db = new DBController();
db.setUp("IT Innovation Project");
String dbQuery = "SELECT * FROM forumTopics WHERE topic_id = '" + id
+ "'";
rs = db.readRequest(dbQuery);
db.terminate();
return success;
}
}
Then, inside the eForumThreadContent, I want to display the content of chosen thread using a table. So I use the id which I pass in just now and insert into my sql statement. Here are my codes.
public eForumThreadContent(int id) {
this.id = id;
}
if (jTable == null) {
Vector columnNames = new Vector(); // Vector class allows dynamic
// array of objects
Vector data = new Vector();
try {
DBController db = new DBController();
db.setUp("IT Innovation Project");
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver").newInstance();
String dsn = "IT Innovation Project";
String s = "jdbc:odbc:" + dsn;
Connection con = DriverManager.getConnection(s, "", "");
String sql = "Select topic_title,topic_description,topic_by from forumTopics WHERE topic_id = '"+id+"'";
java.sql.Statement statement = con.createStatement();
ResultSet resultSet = statement.executeQuery(sql);
ResultSetMetaData metaData = resultSet.getMetaData();
int columns = metaData.getColumnCount();
for (int i = 1; i <= columns; i++) {
columnNames.addElement(metaData.getColumnName(i));
}
while (resultSet.next()) {
Vector row = new Vector(columns);
for (int i = 1; i <= columns; i++) {
row.addElement(resultSet.getObject(i));
}
data.addElement(row);
}
resultSet.close();
((Connection) statement).close();
} catch (Exception e) {
System.out.println(e);
}
jTable = new JTable(data, columnNames);
TableColumn column;
for (int i = 0; i < jTable.getColumnCount(); i++) {
column = jTable.getColumnModel().getColumn(i);
if (i == 1) {
column.setPreferredWidth(400); // second column is bigger
}else {
column.setPreferredWidth(200);
}
}
String header[] = { "Title", "Description", "Posted by" };
for (int i = 0; i < jTable.getColumnCount(); i++) {
TableColumn column1 = jTable.getTableHeader().getColumnModel()
.getColumn(i);
column1.setHeaderValue(header[i]);
}
jTable.getTableHeader().setFont( new Font( "Dialog" , Font.PLAIN, 20 ));
jTable.getTableHeader().setForeground(Color.white);
jTable.getTableHeader().setBackground(new Color(102, 102, 102));
jTable.setEnabled(false);
jTable.setRowHeight(100);
jTable.getRowHeight();
jTable.setFont( new Font( "Dialog" , Font.PLAIN, 18 ));
}
return jTable;
}
However, the table inside eForumThreadContent is empty even when I clicked on certain thread. It also gives me an error message.
[Microsoft][ODBC Microsoft Access Driver] Data type mismatch in criteria expression.
at sun.jdbc.odbc.JdbcOdbc.createSQLException(Unknown Source)
at sun.jdbc.odbc.JdbcOdbc.standardError(Unknown Source)
at sun.jdbc.odbc.JdbcOdbc.SQLExecDirect(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcStatement.execute(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcStatement.executeQuery(Unknown Source)
at DBController.database.DBController.readRequest(DBController.java:27)
at kioskeForum.entity.eForumTopics.retrieveDiscussion(eForumTopics.java:67)
at kioskeForum.ui.eForumDiscussion$3.mouseClicked(eForumDiscussion.java:296)
I research online to get an idea for topic views using id. But my table does not show up anything. Can somebody enlighten me how to fix it? Or any other ways to display topic views in java swing? Thanks in advance.

NHibernate IQueryOver.ToRowCountQuery() equivalent when Using HQL IQuery

I'm trying to implement paging within my project.
When using NHibernate's IQueryOver syntax as shown below things are working as expected.
public PagedResult<T> ExecutePagedQuery(IQueryOver<T,T> query)
{
SetPaging(query);
var results = query.Future<T>();
var count = query.ToRowCountQuery().FutureValue<int>();
return new PagedResult<T>()
{
TotalItemCount = count.Value,
PageOfResults = results.ToList()
};
}
protected virtual void SetPaging(IQueryOver<T, T> query)
{
var maxResults = PageSize;
var numberToSkip = (PageNumber - 1) * PageSize;
query.Skip(numberToSkip).Take(maxResults);
}
For some queries we are using HQL instead of the IQueryOver syntax however.
I'm wondering if there is an equivalent to "query.ToRowCountQuery().FutureValue< int >()" that can be used when querying when passing an IQuery insead of an IQueryOver to provide the row count.
public PagedResult<T> ExecutePagedQuery(IQuery query)
{
SetPaging(query);
var results = query.Future<T>();
var count = // ToRowCountQueryEquivalent?
return new PagedResult<T>()
{
TotalItemCount = count,
PageOfResults = results.ToList()
};
}
protected virtual void SetPaging(IQuery query)
{
var maxResults = PageSize;
var numberToSkip = (PageNumber - 1) * PageSize;
query.SetFirstResult(numberToSkip);
query.SetMaxResults(maxResults);
}
some crazy idea
string countQuery;
if (query.QueryString.StartsWith("SELECT", StringComparison.OrdinalIgnoreCase))
countQuery = Regex.Replace(query.QueryString, "SELECT (.*) FROM", #"SELECT Count($1) FROM", RegexOptions.IgnoreCase);
else
countQuery = "SELECT COUNT(*) " + query.QueryString;
We've since converted our HQL queries to IQueryOver allowing us to take advantage of the ToRowCountQuery() helper.

NHibernate: Saving different types of objects in the same session breaks batching

newbie here, sorry if this is an obvious question.
It seems saving different types of objects in the same session breaks batching, cause significant performance drop.
ID generator is set to Increment (as Diego Mijelshon advised, I tried hilo("100"), but unfortunately same issue, Test1() is still about 5 times slower than Test2()):
public class CustomIdConvention : IIdConvention
{
public void Apply(IIdentityInstance instance)
{
instance.GeneratedBy.Increment();
}
}
AdoNetBatchSize is set to 1000:
MsSqlConfiguration.MsSql2008
.ConnectionString(connectionString)
.AdoNetBatchSize(1000)
.Cache(x => x
.UseQueryCache()
.ProviderClass<HashtableCacheProvider>())
.ShowSql();
These are the models:
public class TestClass1
{
public virtual int Id { get; private set; }
}
public class TestClass2
{
public virtual int Id { get; private set; }
}
These are the test methods. Test1() takes 62 seconds, Test2() takes only 11 seconds. (as Phill advised, I tried stateless sessions, but unfortunately same issue):
[TestMethod]
public void Test1()
{
int count = 50 * 1000;
using (var session = SessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction())
{
for (int i = 0; i < count; i++)
{
var x = new TestClass1();
var y = new TestClass2();
session.Save(x);
session.Save(y);
}
transaction.Commit();
}
}
}
[TestMethod]
public void Test2()
{
int count = 50 * 1000;
using (var session = SessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction())
{
for (int i = 0; i < count; i++)
{
var x = new TestClass1();
session.Save(x);
}
transaction.Commit();
}
}
using (var session = SessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction())
{
for (int i = 0; i < count; i++)
{
var y = new TestClass2();
session.Save(y);
}
transaction.Commit();
}
}
}
Any ideas?
Thanks!
Update:
The test project can be downloaded from here. You need to change the connectionString in the Main method. I changed all sessions to stateless sessions.
My restuls: Test1 = 59.11, Test2 = 7.60, Test3 = 7.72. Test1 is 7.7 times slower than Test2 & Test3!
Do not use increment. It's the worst possible generator.
Try changing it to HiLo.
Update:
It looks like the problem occurs when alternating saves of different entities, regardless of whether the session/transaction are separated or not.
This produces similar results to the second test method:
[TestMethod]
public void Test3()
{
int count = 50 * 1000;
using (var session = SessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction())
{
for (int i = 0; i < count; i++)
{
var x = new TestClass1();
session.Save(x);
}
for (int i = 0; i < count; i++)
{
var y = new TestClass2();
session.Save(y);
}
transaction.Commit();
}
}
}
My guess, without looking at NH's sources, is that it preserves the order because of possible relationships between the entities, even when there are none.
When you run test2 and test3, the insert's are batched together.
When you run test1, where you alternate the inserts, the inserts are issued as separate statements and are not batched together.
I found this out by profiling all three tests.
So as per Diego's answer, it must preserve the order that you're inserting, and batch them together.
I wrote a 4th test, I set the batch size to 10, then alternated when i changed from TestClass1 to TestClass2 so that I was doing 5 of TestClass1 and then 5 of TestClass2, to hit the batch size.
This pushed out batch's of 5 in the order they were processed.
public void Test4()
{
int count = 10;
using (var session = SessionFactory.OpenSession())
using (var transaction = session.BeginTransaction())
{
for (int i = 0; i < count; i++)
{
if (i%2 == 0)
{
for (int j = 0; j < 5; j++)
{
var x = new TestClass1();
session.Save(x);
}
}
else
{
for (int j = 0; j < 5; j++)
{
var y = new TestClass2();
session.Save(y);
}
}
}
transaction.Commit();
}
}
Then I changed it to insert 3 at a time instead of 5. The batch's were in multiples of 3, so what must be happening is the batch size allows a batch of 1 type to go to specified amount, but groups only the same type together. While alternating causes separate insert statements.