Ravendb select - update multiple row - ravendb

I am trying to select and update multiple row from ravendb, but it recursively update same rows. Namely first 100 rows. There is no changes.
Here is my code. How can I select some rows, Update some fields of each rows and do it again and again until my job finished.
var currentEmailId = 100;
using (var session = store.OpenSession())
{
var goon = true;
while(goon){
var contacts = session.Query<Contacts>().Where(f => f.LastEmailId < currentEmailId).Take(100);
if(contacts.Any()){
foreach(var contact in contacts){
EmailOperation.Send(contact, currentEmailId);
contact.LastEmailId = currentEmailId;
}
session.SaveChanges();
}
else{
goon = false
}
}
}

It's probably because you're doing a query immediately after saving changes, without letting the indexes update after save changes. Thus, you're getting back the same items. To fix that, you can tell SaveChanges to wait until indexes are updated. Your code would look something like this:
Try this:
var goon = true;
var currentEmailId = 100;
while (goon)
{
using (var session = store.OpenSession())
{
var contacts = session.Query<Contacts>()
.Where(f => f.LastEmailId < currentEmailId)
.Take(100);
if(contacts.Any())
{
foreach(var contact in contacts)
{
EmailOperation.Send(contact, currentEmailId);
contact.LastEmailId = currentEmailId;
}
// Wait for the indexes to update when calling SaveChanges.
DbSession.Advanced.WaitForIndexesAfterSaveChanges(TimeSpan.FromSeconds(30), false);
session.SaveChanges();
}
else
{
goon = false
}
}
}
If you're updating many contacts at once, you may wish to consider using using Streaming query results combined with BulkInsert to update many Contacts en mass.

Related

Google BigQuery returns only partial table data with C# application using .net Client Library

I am trying to execute the query (Basic select statement with 10 fields). My table contains more than 500k rows. C# application returns the response with only 4260 rows. However Web UI returns all the records.
Why my code returns only partial data, What is the best way to select all the records and load into C# Data Table? If there is any code snippet it would be more helpful to me.
using Google.Apis.Auth.OAuth2;
using System.IO;
using System.Threading;
using Google.Apis.Bigquery.v2;
using Google.Apis.Bigquery.v2.Data;
using System.Data;
using Google.Apis.Services;
using System;
using System.Security.Cryptography.X509Certificates;
namespace GoogleBigQuery
{
public class Class1
{
private static void Main()
{
try
{
Console.WriteLine("Start Time: {0}", DateTime.Now.ToString());
String serviceAccountEmail = "SERVICE ACCOUNT EMAIL";
var certificate = new X509Certificate2(#"KeyFile.p12", "notasecret", X509KeyStorageFlags.Exportable);
ServiceAccountCredential credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(serviceAccountEmail)
{
Scopes = new[] { BigqueryService.Scope.Bigquery, BigqueryService.Scope.BigqueryInsertdata, BigqueryService.Scope.CloudPlatform, BigqueryService.Scope.DevstorageFullControl }
}.FromCertificate(certificate));
BigqueryService Service = new BigqueryService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "PROJECT NAME"
});
string query = "SELECT * FROM [publicdata:samples.shakespeare]";
JobsResource j = Service.Jobs;
QueryRequest qr = new QueryRequest();
string ProjectID = "PROJECT ID";
qr.Query = query;
qr.MaxResults = Int32.MaxValue;
qr.TimeoutMs = Int32.MaxValue;
DataTable DT = new DataTable();
int i = 0;
QueryResponse response = j.Query(qr, ProjectID).Execute();
string pageToken = null;
if (response.JobComplete == true)
{
if (response != null)
{
int colCount = response.Schema.Fields.Count;
if (DT == null)
DT = new DataTable();
if (DT.Columns.Count == 0)
{
foreach (var Column in response.Schema.Fields)
{
DT.Columns.Add(Column.Name);
}
}
pageToken = response.PageToken;
if (response.Rows != null)
{
foreach (TableRow row in response.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
while (true)
{
int StartIndexForQuery = DT.Rows.Count;
Google.Apis.Bigquery.v2.JobsResource.GetQueryResultsRequest SubQR = Service.Jobs.GetQueryResults(response.JobReference.ProjectId, response.JobReference.JobId);
SubQR.StartIndex = (ulong)StartIndexForQuery;
//SubQR.MaxResults = Int32.MaxValue;
GetQueryResultsResponse QueryResultResponse = SubQR.Execute();
if (QueryResultResponse != null)
{
if (QueryResultResponse.Rows != null)
{
foreach (TableRow row in QueryResultResponse.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
if (null == QueryResultResponse.PageToken)
{
break;
}
}
else
{
break;
}
}
}
else
{
Console.WriteLine("Response is null");
}
}
int TotalCount = 0;
if (DT != null && DT.Rows.Count > 0)
{
TotalCount = DT.Rows.Count;
}
else
{
TotalCount = 0;
}
Console.WriteLine("End Time: {0}", DateTime.Now.ToString());
Console.WriteLine("No. of records readed from google bigquery service: " + TotalCount.ToString());
}
catch (Exception e)
{
Console.WriteLine("Error Occurred: " + e.Message);
}
Console.ReadLine();
}
}
}
In this Sample Query get the results from public data set, In table contains 164656 rows but response returns 85000 rows only for the first time, then query again to get the second set of results. (But not known this is the only solution to get all the results).
In this sample contains only 4 fields, even-though it does not return all rows, in my case table contains more than 15 fields, I get response of ~4000 rows out of ~10k rows, I need to query again and again to get the remaining results for selecting 1000 rows takes time up to 2 minutes in my methodology so I am expecting best way to select all the records within single response.
Answer from User #:Pentium10
There is no way to run a query and select a large response in a single shot. You can either paginate the results, or if you can create a job to export to files, then use the files generated in your app. Exporting is free.
Step to run a large query and export results to files stored on GCS:
1) Set allowLargeResults to true in your job configuration. You must also specify a destination table with the allowLargeResults flag.
Example:
"configuration":
{
"query":
{
"allowLargeResults": true,
"query": "select uid from [project:dataset.table]"
"destinationTable": [project:dataset.table]
}
}
2) Now your data is in a destination table you set. You need to create a new job, and set the export property to be able to export the table to file(s). Exporting is free, but you need to have Google Cloud Storage activated to put the resulting files there.
3) In the end you download your large files from GCS.
It my turn to design the solution for better results.
Hoping this might help someone. One could retrieve next set of paginated result using PageToken. Here is the sample code for how to use PageToken. Although, I liked the idea of exporting for free. Here, I write rows to flat file but you could add them to your DataTable. Obviously, it is a bad idea to keep large DataTable in memory though.
public void ExecuteSQL(BigqueryService bqservice, String ProjectID)
{
string sSql = "SELECT r.Dealname, r.poolnumber, r.loanid FROM [MBS_Dataset.tblRemitData] R left join each [MBS_Dataset.tblOrigData] o on R.Dealname = o.Dealname and R.Poolnumber = o.Poolnumber and R.LoanID = o.LoanID Order by o.Dealname, o.poolnumber, o.loanid limit 100000";
QueryRequest _r = new QueryRequest();
_r.Query = sSql;
QueryResponse _qr = bqservice.Jobs.Query(_r, ProjectID).Execute();
string pageToken = null;
if (_qr.JobComplete != true)
{
//job not finished yet! expecting more data
while (true)
{
var resultReq = bqservice.Jobs.GetQueryResults(_qr.JobReference.ProjectId, _qr.JobReference.JobId);
resultReq.PageToken = pageToken;
var result = resultReq.Execute();
if (result.JobComplete == true)
{
WriteRows(result.Rows, result.Schema.Fields);
pageToken = result.PageToken;
if (pageToken == null)
break;
}
}
}
else
{
List<string> _fieldNames = _qr.Schema.Fields.ToList().Select(x => x.Name).ToList();
WriteRows(_qr.Rows, _qr.Schema.Fields);
}
}
The Web UI automatically flattens the data. This means that you see multiple rows for each nested field.
When you run the same query via the API, it won't be flattened, and you get fewer rows, as the nested fields are returned as objects. You should check if this is the case at you.
The other is that indeed you need to paginate through the results. Paging through list results has this explained.
If you want to do only one job, than you should write your query ouput to a table, than export the table as JSON, and download the export from GCS.

NHibernate named query and multiple result sets

We have a stored procedure that returns several tables. When calling it using NHibernate, we use the bean transformer but only get the first table transformed and all other results are ignored.
I know that NH is able to process several queries in one db trip using futures but we only have one query and it produces a result that is similar to what we would get with futures, but getting this from a stored procedure.
I believe this scenario is quite common but could not find any clues. Is it possible to use NH to retrieve such results?
Yes,you can use MultiQuery "Hack" like this:
The procudure:
CREATE PROCEDURE [dbo].[proc_Name]
AS BEGIN
SELECT * FROM Question
SELECT * FROM Question
END
The NHibernate Query Code:
public void ProcdureMultiTableQuery()
{
var session = Session;
var procSQLQuery = session.CreateSQLQuery("exec [proc_Name] ?,?");// prcodure returns two table
procSQLQuery.SetParameter(0, userId);
procSQLQuery.SetParameter(1, page);
procSQLQuery.AddEntity(typeof(Question));
var multiResults = session.CreateMultiQuery()
.Add(procSQLQuery)
// More table your procedure returns,more empty SQL query you should add
.Add(session.CreateSQLQuery(" ").AddEntity(typeof(Question))) // the second table returns Question Model
.List();
if (multiResults == null || multiResults.Count == 0)
{
return;
}
if (multiResults.Count != 2)
{
return;
}
var questions1 = ConvertObjectsToArray<Question>((System.Collections.IList)multiResults[0]);
var questions2 = ConvertObjectsToArray<Question>((System.Collections.IList)multiResults[1]);
}
static T[] ConvertObjectsToArray<T>(System.Collections.IList objects)
{
if (objects == null || objects.Count == 0)
{
return null;
}
var array = new T[objects.Count];
for (int i = 0; i < array.Length; i++)
{
array[i] = (T)objects[i];
}
return array;
}

Bigquery tables.list() returns only 1000 tables?

Am I doing something wrong in the following code or is this a bug?
com.google.api.services.bigquery.Bigquery.Tables.List list =
bigquery.tables().list(PROJECT_ID, datasetid);
list.setMaxResults((long) 5000);
return list.execute().getTables();
I have more than a 1000 tables in this dataset.
The maximum number of tables that will be returned in one request is 1000. However, you should also receive a pageToken in the response that can be used to page through further results.
as in:
List<Table> tables = new ArrayList<>();
com.google.api.services.bigquery.Bigquery.Tables.List list =
bigquery.tables().list(PROJECT_ID, datasetid);
list.setMaxResults(5000L);
String nextPageToken = null;
while (true) {
if (nextPageToken != null) {
list.setPageToken(nextPageToken);
}
TableList result = list.execute();
tables.addAll(result.getTables());
if (result.getNextPageToken() == null) {
break;
} else {
nextPageToken = result.getNextPageToken();
}
}
return tables;

Existing posts keep on re-add upon deletion of selected row in jTable

I try to refresh the data of jTable upon deletion of selected row. Here are my codes to set up table :
private JTable getJTableManageReplies() {
jTableManageReplies.setSelectionMode(ListSelectionModel.SINGLE_SELECTION);
jTableManageReplies.getSelectionModel().addListSelectionListener(
new ListSelectionListener() {
#Override
public void valueChanged(ListSelectionEvent e) {
if (!e.getValueIsAdjusting()) {
int viewRow = jTableManageReplies.getSelectedRow();
// Get the first column data of the selectedrow
int replyID = Integer.parseInt(jTableManageReplies.getValueAt(
viewRow, 0).toString());
eForumRepliesAdmin reply = new eForumRepliesAdmin(replyID);
replyID = JOptionPane.showConfirmDialog(null, "Are you sure that you want to delete the selected reply? " , "Delete replies", JOptionPane.YES_NO_OPTION);
if(replyID == JOptionPane.YES_OPTION){
reply.deleteReply();
JOptionPane.showMessageDialog(null, "Reply has been deleted successfully.");
SetUpJTableManageReplies();
}
}
}
});
return jTableManageReplies;
}
public void SetUpJTableManageReplies() {
DefaultTableModel tableModel = (DefaultTableModel) jTableManageReplies
.getModel();
String[] data = new String[5];
db.setUp("IT Innovation Project");
String sql = "Select forumReplies.reply_ID,forumReplies.reply_topic,forumTopics.topic_title,forumReplies.reply_content,forumReplies.reply_by from forumReplies,forumTopics WHERE forumReplies.reply_topic = forumTopics.topic_id ";
ResultSet resultSet = null;
resultSet = db.readRequest(sql);
jTableManageReplies.repaint();
tableModel.getDataVector().removeAllElements();
try {
while (resultSet.next()) {
data[0] = resultSet.getString("reply_ID");
data[1] = resultSet.getString("reply_topic");
data[2] = resultSet.getString("topic_title");
data[3] = resultSet.getString("reply_content");
data[4] = resultSet.getString("reply_by");
tableModel.addRow(data);
}
resultSet.close();
} catch (Exception e) {
System.out.println(e);
}
}
And this is my sql statement :
public boolean deleteReply() {
boolean success = false;
DBController db = new DBController();
db.setUp("IT Innovation Project");
String sql = "DELETE FROM forumReplies where reply_ID = " + replyID
+ "";
if (db.updateRequest(sql) == 1)
success = true;
db.terminate();
return success;
}
I called the repaint() to update the table data with the newest data in database and it works. I mean the data after deletion of certain row. However, the existing posts will keep on re-add. Then I add the removeAllElement method to remove all the existing posts because my sql statement is select * from table. Then, there is an error message which is ArrayIndexOutOfBoundsException. Any guides to fix this? Thanks in advance.
I called the repaint() to update the table data with the newest data
in database and it works.
There is no need to call repaint method when data is changed. Data change is handled by the Table Model (DefaultTableModel in this case.) And fireXXXMethods are required to be called whenever data is changed but you are using DefaultTableModel even those are not required. (Since by default it call these methods when ever there is a change.)
I think the problem is in the valuesChanged(..) method. You are getting the value at row 0 but not checking whether table has rows or not. So keep a constraint.
int viewRow = jTableManageReplies.getSelectedRow();
// Get the first column data of the selectedrow
if(jTableManageReplies.getRowCount() > 0)
int replyID = Integer.parseInt(jTableManageReplies.getValueAt(viewRow, 0).toString());

How can I do this all in one query? Nhibernate

I have a list of Ids and I want to get all the rows back in one query. As a list of objects(So a List of Products or whatever).
I tried
public List<TableA> MyMethod(List<string> keys)
{
var query = "SELECT * FROM TableA WHERE Keys IN (:keys)";
var a = session.CreateQuery(query).SetParameter("keys", keys).List();
return a; // a is a IList but not of TableA. So what do I do now?
}
but I can't figure out how to return it as a list of objects. Is this the right way?
List<TableA> result = session.CreateQuery(query)
.SetParameterList("keys", keys)
.List<TableA>();
Howeever there could be a limitation in this query if number of ":keys" exceed more than 1000 (incase of oracle not sure with other dbs) so i would recommend to use ICriteria instead of CreateQuery- native sqls.
Do something like this,
[TestFixture]
public class ThousandIdsNHibernateQuery
{
[Test]
public void TestThousandIdsNHibernateQuery()
{
//Keys contains 1000 ids not included here.
var keys = new List<decimal>();
using (ISession session = new Session())
{
var tableCirt = session.CreateCriteria(typeof(TableA));
if (keys.Count > 1000)
{
var listsList = new List<List<decimal>>();
//Get first 1000.
var first1000List = keys.GetRange(0, 1000);
//Split next keys into 1000 chuncks.
for (int i = 1000; i < keys.Count; i++)
{
if ((i + 1)%1000 == 0)
{
var newList = new List<decimal>();
newList.AddRange(keys.GetRange(i - 999, 1000));
listsList.Add(newList);
}
}
ICriterion firstExp = Expression.In("Key", first1000List);
ICriterion postExp = null;
foreach (var list in listsList)
{
postExp = Expression.In("Key", list);
tableCirt.Add(Expression.Or(firstExp, postExp));
firstExp = postExp;
}
tableCirt.Add(postExp);
}
else
{
tableCirt.Add(Expression.In("key", keys));
}
var results = tableCirt.List<TableA>();
}
}
}