I'm getting a vague error back from AWS Keyspaces that I'm trying to troubleshoot, but it looks as though AWS only stores DDL statement logs for Keyspaces in Cloudtrail, and not DML. I can see the 'error count' in my dashboard in the AWS Console for Keyspaces, but no details about the errors themselves. This is NOT a question about what might be wrong with my insert statements, or otherwise, and only a question about whether these errors are recorded somewhere in AWS.
I understand that there are other ways to approach this from a troubleshooting standpoint, but I want to first establish whether DML related server-side errors are logged by AWS somewhere, or could if they could be, given extra configuration.
In case someone has encountered something similar, here is all the detail I'm given back from the error itself:
com.datastax.oss.driver.api.core.servererrors.ServerError: java.lang.AssertionError
Do you have more of a stack trace?
You can improve observability with these cloudformation "cloudwatch" templates. Provide the keyspace or table and the template with generate some dashboards for you.
https://github.com/aws-samples/amazon-keyspaces-cloudwatch-cloudformation-templates
Related
In gcp, I need to update a bigquery table whenever a file (multiple formats such as json,xml) gets uploaded into a bucket. I have two options but not sure what are the pros/cons of each of them. Can someone suggest which is a better solution and why?
Approach 1 :
File uploaded to bucket --> Trigger Cloud Function (which updates the bigquery table) -->Bigquery
Approach 2:
File uploaded to bucket --> Trigger Cloud Function (which triggers a dataflow job) -->Dataflow-->Bigquery.
In production env, which approach is better suited and why? If there are alternative approaches,pls let me know.
This is quite a broad question, so I wouldn't be surprised if it gets voted to be closed. That said however, I'd always go #2 (GCS -> CF -> Dataflow -> BigQuery).
Remember, with Cloud Funtions there is a max execution time. If you kick off a load job from the Cloud Function, you'll need to bake logic into it to poll and check the status (load jobs in BigQuery are async). If it fails, you'll need to handle it. But, what if it's still running and you hit the max execution of your Cloud Function?
At least by using Dataflow, you don't have the problem of max execution times and you can simply rerun your pipeline if it fails for some transient reason e.g. network issues.
For the BQ team, queries that usually work, are failing sometimes.
Could you please look into what could be the issue, there is just this:
Error: Unexpected. Please try again.
Job ID: aerobic-forge-504:job_DTFuQEeVGwZbt-PRFbMVE6TCz0U
Sorry for the slow response.
BigQuery replicates data to multiple datacenters and may run queries from any of them. Usually this is transparent ... if your data hasn't replicated everywhere, we will try to find a datacenter that has all of the data necessary to run a query and execute the query from there.
The error you hit was due to BigQuery not being able to find a single datacenter that had all of the data for one of your tables. We try very hard to make sure this doesn't happen. In principle, it should be very rare (we've got solution designed to make sure that it never happens, but haven't finished the implementation yet). We saw an uptick in this issue this morning, and have a bug filed and are currently investigating the issue.
Was this a transient error? If you retry the operation, does it work now? Are you still getting errors on other queries?
a friend asked me if there is a way to see the past dml statements and I wasn't really sure on how to go about answering that question. What he wants to see is the last set of insert statements. So that means it could be more than 1 record. At first I was just saying to check the latest identity, but then he asked what if more inserts were performed at the same time. Can you guys help me out? Is there a DMV I should use that I just don't know about? Thanks.
If you did not prepare for this question then there is no build in way to get to that information. However, you could use third party log reader tools to recover (all) the last statements that where executed against the database. This requires the database to be in Full recovery mode. You could potentially go back as far as you have log backups with this method.
If you want to prepare for that question being ask in the future, you have several options.
The most obvious one is Change Data Capture. You also could write a trigger yourself that records data changes.
You could also run a trace capturing SQL Batch Started.
Finally you could use a third party network sniffer/logger to capture all statements send to the server (this however requires that connection encryption is not used).
I am trying to insert some values into an SQL database, but continuously receiving an error that "Transaction Log for database is full".
I checked it online and some people have suggested to commit work more frequently. I have done than with no changes.
I don't have authorization to change database server. Is this problem client-side or due to server? If due to client, what is the solution?
You are probably getting the error code SQL0964, aren't you?
Probably you execute your DML sentences when there is a lot of activity in the database, and you got this message because your sentences overpassed the log size limit.
The concurrent transactions have used all the available logs (primary and secondary) and your sentences cannot be written in the logs.
Also, if the database has archive log method, probably the file system is full and the database cannot archive the active logs, blocking you to write your sentences in the transaction logs.
Any of these situations can be solved by your DBA team. You can only change the way you insert the rows by doing commit more frequently (if currently a commit is each 500 rows, try a commit each 100 rows, something like that)
For more information about the problem: http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/topic/com.ibm.db2.luw.messages.sql.doc/doc/msql00964c.html
It's a server side issue, you should forward the messages to the DBA team.
I have a Stored Procedure that is constantly failing with the error message "Timeout expired," on a specific user.
All other users are able to invoke the sp just fine, and even I am able to invoke the sp normally using the Query Analyzer--it finishes in just 10 seconds. However with the user in question, the logs show that the ASP always hangs for about 5 minutes and then aborts with a timeout.
I invoke from the ASP page like so "EXEC SP_TV_GET_CLOSED_BANKS_BY_USERS '006111'"
Anybody know how to diagnose the problem? I have already tried looking at deadlocks in the DB, but didn't find any.
Thanks,
Some thoughts...
Reading the comments suggests that parameter sniffing is causing the issue.
For the other users, the cached plan is good enough for the parameter that they send
For this user, the cached plan is probably wrong
This could happen if this user has far more rows than other users, or has rows in another table (so a different table/index seek/scan would be better)
To test for parameter sniffing:
use RECOMPILE (temporarily) on the call or in the def. This could be slow for complex query
Rebuild the indexes (or just statistics) after the timeout and try again. This invalidates all cached plans
To fix:
Mask the parameter
DECLARE #MaskedParam varchar(10)
SELECT #MaskedParam = #SignaureParam
SELECT...WHERE column = #MaskedParam
Just google "Parameter sniffing" and "Parameter masking"
I think to answer your question, we may need a bit more information.
For example, are you using Active directory to authenticate your users? Have you used the SQL profiler to investigate? It sounds like it could be an auth issue where SQL Server is having problems authenticating this particular user.
Sounds to me like a dead lock issue..
Also make sure this user has execute rights and read rights in SQL Server
But if at the time info is being written as its trying to be read you will dead lock, as the transaction has not yet been committed.
Jeff did a great post about his experience with that and stackoverflow.
http://www.codinghorror.com/blog/archives/001166.html
Couple of things to check:
Does this happen only on that specific user's machine? Can he try it from another
machine? - it might be a client configuration problem.
Can you capture the actual string that this specific user runs and run it from an ASP page? It might be that user executes the SP in a way that generates either a loop or a massive load of data.
Finally, if you're using an intra-organization application, it might be that your particular user's permissions are different than the others. You can compare them at the Active Directory level.
Now, I can recommend a commercial software that will definitely solve your issue. It records end-to-end transactions, and analyzes particular failures. But I do not want to advertise in this forum. If you'd like, drop me a note and I'll explain more.
Well, I could suggest that you use SQL Server Profiler and open a new session. Invoke your stored procedure from your ASP page and see what is happening. While this may not solve your problem, it can surely provide a starting point for you to carry out some 'investigation' of your own.