I have a following sql code, if I set AutoDetectChangesEnabled and ValidateOnSaveEnabled to false. When there is a race condition in database, is it possible that my string process is processing half way then exit the using.
using (DbContext dbContext = new DbContext()) {
dbContext.Configuration.AutoDetectChangesEnabled = false;
dbContext.Configuration.ValidateOnSaveEnabled = false;
string a = dbContext.sp_getsomething().FirstOrDefault();
string b = a.substring(2);
//and other string processing
}
Related
I have a C# .net application using the rally 3.0.1 API. When I query task in my system I get 0.0 for time spent when I know they have time against them. Anyone know how to get this? Below is my code:
if (uTasks.Count > 0)
{
Request taskRequest = new Request(resultChild["Tasks"]);
QueryResult TaskQueryResult = restApi.Query(taskRequest);
foreach (var items in TaskQueryResult.Results)
//foreach (var items in uTasks)
{
DataRow dtrow2;
dtrow2 = dt.NewRow();
dtrow2["TaskID"]=items["FormattedID"];
dtrow2["Task Name"] = items["Name"];
if (items["Owner"] != null)
{
var owner = items["Owner"];
String ownerref = owner["_ref"];
var ownerFetch = restApi.GetByReference(ownerref, "Name");
string strTemp = ownerFetch["_refObjectName"];
dtrow2["Owner"] = strTemp.Replace(",", " ");
}
\\else { dtrow2["Owner"] = ""; }
dtrow2["Task-Est"] = items["Estimate"];
dtrow2["Task-ToDo"] = items["ToDo"];
dtrow2["Task-Spent"] = items["TimeSpent"];
dtrow2["ObjectType"] = "T";
dt.Rows.Add(dtrow2);
}
}
It seems like that should work. You may want to make sure you're including the TimeSpent field in your fetch before issuing the request.
taskRequest.Fetch = new List<string>() { "TimeSpent" };
I am trying to execute the query (Basic select statement with 10 fields). My table contains more than 500k rows. C# application returns the response with only 4260 rows. However Web UI returns all the records.
Why my code returns only partial data, What is the best way to select all the records and load into C# Data Table? If there is any code snippet it would be more helpful to me.
using Google.Apis.Auth.OAuth2;
using System.IO;
using System.Threading;
using Google.Apis.Bigquery.v2;
using Google.Apis.Bigquery.v2.Data;
using System.Data;
using Google.Apis.Services;
using System;
using System.Security.Cryptography.X509Certificates;
namespace GoogleBigQuery
{
public class Class1
{
private static void Main()
{
try
{
Console.WriteLine("Start Time: {0}", DateTime.Now.ToString());
String serviceAccountEmail = "SERVICE ACCOUNT EMAIL";
var certificate = new X509Certificate2(#"KeyFile.p12", "notasecret", X509KeyStorageFlags.Exportable);
ServiceAccountCredential credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(serviceAccountEmail)
{
Scopes = new[] { BigqueryService.Scope.Bigquery, BigqueryService.Scope.BigqueryInsertdata, BigqueryService.Scope.CloudPlatform, BigqueryService.Scope.DevstorageFullControl }
}.FromCertificate(certificate));
BigqueryService Service = new BigqueryService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "PROJECT NAME"
});
string query = "SELECT * FROM [publicdata:samples.shakespeare]";
JobsResource j = Service.Jobs;
QueryRequest qr = new QueryRequest();
string ProjectID = "PROJECT ID";
qr.Query = query;
qr.MaxResults = Int32.MaxValue;
qr.TimeoutMs = Int32.MaxValue;
DataTable DT = new DataTable();
int i = 0;
QueryResponse response = j.Query(qr, ProjectID).Execute();
string pageToken = null;
if (response.JobComplete == true)
{
if (response != null)
{
int colCount = response.Schema.Fields.Count;
if (DT == null)
DT = new DataTable();
if (DT.Columns.Count == 0)
{
foreach (var Column in response.Schema.Fields)
{
DT.Columns.Add(Column.Name);
}
}
pageToken = response.PageToken;
if (response.Rows != null)
{
foreach (TableRow row in response.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
while (true)
{
int StartIndexForQuery = DT.Rows.Count;
Google.Apis.Bigquery.v2.JobsResource.GetQueryResultsRequest SubQR = Service.Jobs.GetQueryResults(response.JobReference.ProjectId, response.JobReference.JobId);
SubQR.StartIndex = (ulong)StartIndexForQuery;
//SubQR.MaxResults = Int32.MaxValue;
GetQueryResultsResponse QueryResultResponse = SubQR.Execute();
if (QueryResultResponse != null)
{
if (QueryResultResponse.Rows != null)
{
foreach (TableRow row in QueryResultResponse.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
if (null == QueryResultResponse.PageToken)
{
break;
}
}
else
{
break;
}
}
}
else
{
Console.WriteLine("Response is null");
}
}
int TotalCount = 0;
if (DT != null && DT.Rows.Count > 0)
{
TotalCount = DT.Rows.Count;
}
else
{
TotalCount = 0;
}
Console.WriteLine("End Time: {0}", DateTime.Now.ToString());
Console.WriteLine("No. of records readed from google bigquery service: " + TotalCount.ToString());
}
catch (Exception e)
{
Console.WriteLine("Error Occurred: " + e.Message);
}
Console.ReadLine();
}
}
}
In this Sample Query get the results from public data set, In table contains 164656 rows but response returns 85000 rows only for the first time, then query again to get the second set of results. (But not known this is the only solution to get all the results).
In this sample contains only 4 fields, even-though it does not return all rows, in my case table contains more than 15 fields, I get response of ~4000 rows out of ~10k rows, I need to query again and again to get the remaining results for selecting 1000 rows takes time up to 2 minutes in my methodology so I am expecting best way to select all the records within single response.
Answer from User #:Pentium10
There is no way to run a query and select a large response in a single shot. You can either paginate the results, or if you can create a job to export to files, then use the files generated in your app. Exporting is free.
Step to run a large query and export results to files stored on GCS:
1) Set allowLargeResults to true in your job configuration. You must also specify a destination table with the allowLargeResults flag.
Example:
"configuration":
{
"query":
{
"allowLargeResults": true,
"query": "select uid from [project:dataset.table]"
"destinationTable": [project:dataset.table]
}
}
2) Now your data is in a destination table you set. You need to create a new job, and set the export property to be able to export the table to file(s). Exporting is free, but you need to have Google Cloud Storage activated to put the resulting files there.
3) In the end you download your large files from GCS.
It my turn to design the solution for better results.
Hoping this might help someone. One could retrieve next set of paginated result using PageToken. Here is the sample code for how to use PageToken. Although, I liked the idea of exporting for free. Here, I write rows to flat file but you could add them to your DataTable. Obviously, it is a bad idea to keep large DataTable in memory though.
public void ExecuteSQL(BigqueryService bqservice, String ProjectID)
{
string sSql = "SELECT r.Dealname, r.poolnumber, r.loanid FROM [MBS_Dataset.tblRemitData] R left join each [MBS_Dataset.tblOrigData] o on R.Dealname = o.Dealname and R.Poolnumber = o.Poolnumber and R.LoanID = o.LoanID Order by o.Dealname, o.poolnumber, o.loanid limit 100000";
QueryRequest _r = new QueryRequest();
_r.Query = sSql;
QueryResponse _qr = bqservice.Jobs.Query(_r, ProjectID).Execute();
string pageToken = null;
if (_qr.JobComplete != true)
{
//job not finished yet! expecting more data
while (true)
{
var resultReq = bqservice.Jobs.GetQueryResults(_qr.JobReference.ProjectId, _qr.JobReference.JobId);
resultReq.PageToken = pageToken;
var result = resultReq.Execute();
if (result.JobComplete == true)
{
WriteRows(result.Rows, result.Schema.Fields);
pageToken = result.PageToken;
if (pageToken == null)
break;
}
}
}
else
{
List<string> _fieldNames = _qr.Schema.Fields.ToList().Select(x => x.Name).ToList();
WriteRows(_qr.Rows, _qr.Schema.Fields);
}
}
The Web UI automatically flattens the data. This means that you see multiple rows for each nested field.
When you run the same query via the API, it won't be flattened, and you get fewer rows, as the nested fields are returned as objects. You should check if this is the case at you.
The other is that indeed you need to paginate through the results. Paging through list results has this explained.
If you want to do only one job, than you should write your query ouput to a table, than export the table as JSON, and download the export from GCS.
I have the following code where the user is prompted to enter his username and password in a form. The username and password are checked with the database and if correct the user is logged in. However this code can be easily SQL injected for example by typing:
UserName = 'x' and UserPwd = 'x' or 'x'
Can someone help me to modify the code to prevent SQL injection. Here is the code:
<%#LANGUAGE=Jscript%>
<%
// ----- GLOBALS DECLARATIONS ----------------------------------------------------------------------------
var CKEDir = "ckeditor/";
var DB = Server.MapPath("DB/CMS.MDB");
// ----- GENERAL PURPOSE FUNCTIONS -----------------------------------------------------------------------
// Uses regular expressions to change all single quotes in a string to the HTML
// entity ' and replaces all carriage return and newline characters to spaces.
// This ensures that the string can be incorporated in a SQL statement.
function cleanString(s) {
s = s.replace(/'/g, "'"); // SO syntax fix '
s = s.replace(/[\r\n]/g,' ');
return s;
}
// ----- DATABASE FUNCTIONS ------------------------------------------------------------------------------
// Creates a connection to the database named in the parameter,
function getDBConnection() {
var DBCon = Server.CreateObject("ADODB.Connection");
var DBasePath = DB;
var ConStr = "Provider=Microsoft.Jet.OLEDB.4.0; Data Source=" + DBasePath + ";Persist Security Info=False";
DBCon.Open(ConStr,"","");
return DBCon;
}
// Increments counter for current page (as identified by global variable PageID) in
// table Counters, and returns a string indicating number of times page was accessed.
function getAccess() {
var msg = '';
if (PageID) {
var DBConn = getDBConnection();
var Td = new Date();
var SQL = "SELECT * FROM Counters WHERE PageID=" + PageID ;
var RS = DBConn.Execute(SQL);
// Page counter does not yet exist - create it.
if (RS.Eof)
{
var AccessCount=1;
var AccessSince = new Date();
SQL="INSERT into Counters ([PageID]) VALUES ("+PageID+")";
}
// Page counter exists, increment it.
else
{
var AccessCount=RS("Hits")+1;
var AccessSince=RS("Created").value;
SQL="UPDATE Counters SET [Hits]="+AccessCount+" WHERE [PageID]="+PageID;
}
RS = DBConn.Execute(SQL)
DBConn.Close();
msg = AccessCount + " visits since " + AccessSince;
}
return msg;
}
// ----- LOGGING IN AND OUT FUNCTIONS --------------------------------------------------------------------
// Returns true if user is logged in.
function isLoggedIn() {
return Session("UserID");
}
// Checks given name and password in users database.
// No validation on the user input is performed, so this function is
// susceptible to SQL injection attacks.
function logInUser(name,pwd) {
var DBConn = getDBConnection();
var SQL = "SELECT * FROM Users WHERE UserName = '" + name + "' and UserPwd = '" + pwd + "'";
var RS = DBConn.Execute(SQL);
var valid = !RS.Eof;
if (valid) {
Session("UserID") = RS("UserID").value;
Session("UserName") = RS("UserName").value;
Session("UserFullName") = RS("UserFirstName").value + ' ' + RS("UserLastName").value;
}
DBConn.Close;
return valid;
}
// Logs out current user.
function logOutUser() {
Session("UserID") = 0;
}
// Returns full name of currently logged in user if any.
function loggedInUser() {
var msg = '';
if (Session("UserID")) msg = Session("UserFullName");
return msg;
}
// Returns true if current user can edit content.
// Currently allows any authenticated user to edit content.
function inEditMode() {
return isLoggedIn();
}
%>
Use parameterized queries. It prevents the SQL injections.
Click here for more documentation
It will prevent the SQL string being hijacked by malicious input.
Good luck!
I have a pivot table, and I am trying to select certain pivot items based on values in an array. I need this process to go faster, so I have tried using application.calculation = xlcalculationmanual and pivottables.manualupdate = true, but neither seem to be working; the pivot table still recalculates each time I change a pivot item.
Is there something I can do differently to prevent Excel from recalculating each time?
Here is my code:
Application.Calculation = xlCalculationManual
'code to fill array with list of companies goes here
dim PT As Excel.PivotTable
Set PT = Sheets("LE Pivot Table").PivotTables("PivotTable1")
Sheets("LE Pivot Table").PivotTables("PivotTable1").ManualUpdate = True
dim pivItem As PivotItem
'compare pivot items to array. If pivot item matches an element of the array, make it visible=true, otherwise, make it visible=false
For Each pivItem In PT.PivotFields("company").PivotItems
pivItem.Visible = False 'initially make item unchecked
For Each company In ArrayOfCompanies()
If pivItem.Value = company Then
pivItem.Visible = True
End If
Next company
Next pivItem
pivottable.ManualUpdate [ = setting ]
True causes RefreshTable to clear data from the pivot table, rather than refreshing it
False allows RefreshTable to work normally.
Default is False.
This property is reset to False automatically after the calling procedure ends (important)
This property should be set to true just before you make an update (e.g. changing pivot item Visible property)
Below is some code written in C# as an example:
private void FilterByPivotItems(PivotField pf, List<string> pivotItemNames)
{
PivotItems pis = pf.ChildItems;
if (pf.Orientation != 0)
{
int oldAutoSortOrder = 0;
if (pf.AutoSortOrder != (int)Constants.xlManual)
{
oldAutoSortOrder = pf.AutoSortOrder;
pf.AutoSort((int)Constants.xlManual, pf.Name);
}
int pivotItemsCount = pf.PivotItems().Count;
for (int i = 1; i <= pivotItemsCount; i++)
{
PivotItem pi = pf.PivotItems(i);
// check if current pivot item needs to be hidden (if it exists in pivotItemNames)
var match = pivotItemNames.FirstOrDefault(stringToCheck => stringToCheck.Equals(pi.Value));
if (match == null)
{
TryFilterPivotItems(pi, false, true);
}
else
{
TryFilterPivotItems(pi, true, true);
}
}
if (oldAutoSortOrder != 0)
{
pf.AutoSort(oldAutoSortOrder, pf.Name);
}
PivotTable pt = pf.Parent as PivotTable;
if (pt != null)
{
// changing a pivot item triggers pivot table update
// so a refresh should be avoided cause it takes a lot and is unnecessary in this case
pt.Update();
}
}
}
private void TryFilterPivotItems(PivotItem currentPI, bool filterValue, bool deferLayoutUpdate = false)
{
try
{
PivotField pf = currentPI.Parent;
PivotTable pt = pf.Parent as PivotTable;
if (currentPI.Visible != filterValue)
{
if (deferLayoutUpdate == true && pt != null)
{
// just keep these three lines stick together, no if, no nothing (otherwise ManualUpdate will reset back to false)
pt.ManualUpdate = true;
currentPI.Visible = filterValue;
// this may be redundant since setting Visible property of pivot item, resets ManualUpdate to false
pt.ManualUpdate = false;
}
else
{
currentPI.Visible = filterValue;
}
}
}
catch (Exception ex)
{
}
}
private void TryFilterPivotItems(PivotField pf, string itemValue, bool filterValue, bool deferLayoutUpdate = false)
{
try
{
PivotItem currentPI = pf.PivotItems(itemValue);
TryFilterPivotItems(currentPI, filterValue, deferLayoutUpdate);
}
catch (Exception ex)
{
}
}
As a conclusion, ManualUpdate property change doesn't stay for long (in my tests, I could see that it gets reset to false as soon as possible, so that's why I recommended you to set it to true whenever you want to make a change for a pivot item)
For more info on what means an update in Excel, you can check the following:
Pivot Refresh vs. Update – is there a real difference?
References:
Title: Programming Excel with VBA and .NET
By: Jeff Webb, Steve Saunders
Print ISBN: 978-0-596-00766-9 | ISBN 10: 0-596-00766-3
Ebook ISBN: 978-0-596-15951-1 | ISBN 10: 0-596-15951-X
I try to refresh the data of jTable upon deletion of selected row. Here are my codes to set up table :
private JTable getJTableManageReplies() {
jTableManageReplies.setSelectionMode(ListSelectionModel.SINGLE_SELECTION);
jTableManageReplies.getSelectionModel().addListSelectionListener(
new ListSelectionListener() {
#Override
public void valueChanged(ListSelectionEvent e) {
if (!e.getValueIsAdjusting()) {
int viewRow = jTableManageReplies.getSelectedRow();
// Get the first column data of the selectedrow
int replyID = Integer.parseInt(jTableManageReplies.getValueAt(
viewRow, 0).toString());
eForumRepliesAdmin reply = new eForumRepliesAdmin(replyID);
replyID = JOptionPane.showConfirmDialog(null, "Are you sure that you want to delete the selected reply? " , "Delete replies", JOptionPane.YES_NO_OPTION);
if(replyID == JOptionPane.YES_OPTION){
reply.deleteReply();
JOptionPane.showMessageDialog(null, "Reply has been deleted successfully.");
SetUpJTableManageReplies();
}
}
}
});
return jTableManageReplies;
}
public void SetUpJTableManageReplies() {
DefaultTableModel tableModel = (DefaultTableModel) jTableManageReplies
.getModel();
String[] data = new String[5];
db.setUp("IT Innovation Project");
String sql = "Select forumReplies.reply_ID,forumReplies.reply_topic,forumTopics.topic_title,forumReplies.reply_content,forumReplies.reply_by from forumReplies,forumTopics WHERE forumReplies.reply_topic = forumTopics.topic_id ";
ResultSet resultSet = null;
resultSet = db.readRequest(sql);
jTableManageReplies.repaint();
tableModel.getDataVector().removeAllElements();
try {
while (resultSet.next()) {
data[0] = resultSet.getString("reply_ID");
data[1] = resultSet.getString("reply_topic");
data[2] = resultSet.getString("topic_title");
data[3] = resultSet.getString("reply_content");
data[4] = resultSet.getString("reply_by");
tableModel.addRow(data);
}
resultSet.close();
} catch (Exception e) {
System.out.println(e);
}
}
And this is my sql statement :
public boolean deleteReply() {
boolean success = false;
DBController db = new DBController();
db.setUp("IT Innovation Project");
String sql = "DELETE FROM forumReplies where reply_ID = " + replyID
+ "";
if (db.updateRequest(sql) == 1)
success = true;
db.terminate();
return success;
}
I called the repaint() to update the table data with the newest data in database and it works. I mean the data after deletion of certain row. However, the existing posts will keep on re-add. Then I add the removeAllElement method to remove all the existing posts because my sql statement is select * from table. Then, there is an error message which is ArrayIndexOutOfBoundsException. Any guides to fix this? Thanks in advance.
I called the repaint() to update the table data with the newest data
in database and it works.
There is no need to call repaint method when data is changed. Data change is handled by the Table Model (DefaultTableModel in this case.) And fireXXXMethods are required to be called whenever data is changed but you are using DefaultTableModel even those are not required. (Since by default it call these methods when ever there is a change.)
I think the problem is in the valuesChanged(..) method. You are getting the value at row 0 but not checking whether table has rows or not. So keep a constraint.
int viewRow = jTableManageReplies.getSelectedRow();
// Get the first column data of the selectedrow
if(jTableManageReplies.getRowCount() > 0)
int replyID = Integer.parseInt(jTableManageReplies.getValueAt(viewRow, 0).toString());