I ran a hive query on a decently large dataset and it was taking too much time for the query so I decided to kill the application with :
yarn kill -application-id
Now when I check from the CLI with:
yarn application -list
then the above mentioned application does not show up in the list.
However, when I log into the Tez view from ambari the application is showing up to be still in the running state(almost been 24 hours since I created it).
I tried killing it again from the command line but it says that the application has already finished.
I also checked in the resource manager UI and the status for that job shows that it was killed.
Because of this, whenever I am trying to run any new hive job, it is just getting queued up and I am unable to run any other jobs.
Please help!
The TEZ VIEW is an export of the application timeline server info. If you use yarn kill, hive does not properly inform the YARN Application Timeline Server that the query has been terminated. Therefore, you still see these as running in the tez view because ATS never received any update that this entered a stopped/failed state. If you are unable to run new hive jobs, it will not be related to the fact that killed applications still show as running in the tez view and you should troubleshoot that separately. The bug you described is purely cosmetic and is documented in the following places:
https://issues.apache.org/jira/browse/HIVE-16429
https://community.hortonworks.com/content/supportkb/196542/tez-ui-displays-query-as-running-even-after-a-succ.html
So the way around I found to clear the queue so that I could run other queries was to go to /hadoop/yarn/timeline, backup the files and restart YARN. The TEZ queue was cleared up and I could start running my queries from the hive view again.
I should mention, however, that this will clear all queries(for all the users).
Related
I have an SSIS package that uses the Execute Process Task to run an application. Running on my local visual studio has no issues. Running it on the server as a job, from the logs, I can see that it just hangs on the EPT step. It never errors or stops.
The old version of the application runs from the same directory as part of a job on a server with no issue. I'm simply just running a new version of the application. It does pass arguments slightly different though.
I log into the server as the SQL Agent service account and run the executable commands from CMD the same as the SSIS package runs it, and it runs fine.
The task is configured with default configurations except for the window is hidden, the same as the old version that works.
I've seen many people have this issue and have tried everything..
I'm assuming that it's prompting me something when it runs on the server. Is there any way to see what it's prompting if I am logged into the server running the job?
I'm using GraphDB Free 8.4.1 in research project and sometimes it gets stucked without any obvious reason. Process is running, but it is not possible to connect to repository, not even from workbench. It just stops responding.
I have to kill the GraphDB and then run it again.
I'm aware, that free edition allows only 2 queries in parallel, but i would not expect it would get stucked. I tried to reproduce this by making 30 parallel SPARQL queries, but everything worked ok.
There is no exception, nothing in error log, nothing in main log.
Please, where could be the problem?
Thank you very much in advance.
You can diagnose the problem by running jstack on the GraphDB process. This will give us details about the process activity.
We have a task where we need to automatically convert an excel file to a csv to prep it for loading into a SQL database. The developers built this process into a SSIS package. For the conversion, they initially tried to have a task in the SSIS package execute a VBscript to convert the file. When they were running this on there local machines, this worked correctly. When they ran the package manually through VS on the server, it ran correctly. When they ran the package manually via the Integration Catalog it ran correctly. We did this both as our accounts and as the service account and got the same results. However, when we scheduled it as a job it would hang on the part of the process that executed the VBScript. No errors, it would just hang until you killed the job.
The job was executing as the service account which has full admin access on the server, explicit full access to the share where the files are stored and converted (which is on the same server) and full admin access to SQL. The job owner is set to sa which uses the service account. And all the job does is execute the package from the integration catalog which works if you run it independently of a job. When we compared the ssisdb execution report for the manual run in the integration catalog using the service account to the job run they looked the same except the job hung on the conversion task and the other did not.
After spending some time trying to figure this out, the developers tried a different solution. They changed the conversion script from using VBScript to using C#. They ran the package from there local machine and once again the package worked. This time when they ran it manually on the server it failed. When we ran it from the integration catalog it failed and when we ran it from a job it failed.
The error we keep getting is "Create CSV file: Error: Exception has been thrown by the target of an invocation" After spending several hours looking into this error nothing suggested seems to be working.
We also tried these same solutions on a newly built server to make sure we weren't dealing with an odd configuration setting that could have been changed (It is a Dev server) and it still failed there.
At this point, we are pretty lost at what is happening. The first solution worked, but for some reason would never work as a job. The 2nd solution works everywhere except when ran on the server.
We are looking at some other solutions to try to get around this. The next thing may be trying to using powershell to convert the file, but not sure if that will bring us back to the same issue. Any suggestion you guys have will be greatly appreciated
Also, we are using SQL Server 2012 dev edition, VS 2012 pro, Windows Server 2012 R2
This might be because of a bug that Excel has when trying to run jobs (that use Excel) and no user is logged on a specific machine. This might affect also the excel library. The solution is to create the following 2 folders:
C:\Windows\SysWOW64\config\systemprofile\Desktop
C:\Windows\System32\config\systemprofile\Desktop
and then restart the machine. :)
(Edited to show that a restart is needed. thanks for Dustin for checking this)
We have Couchbase 4.0.x cluster running on Windows with multiple buckets. I have set up full backups (via Task Scheduler) via Powershell script, it worked ok for about about 2 weeks and then I noticed something strange - cbbackup would just freeze in a middle of a task and won't move forward. I have a log that says it stops at different point every time. The bucket in question isn't even big - less than 10Gb on disk.
About my script - all it does is call cbbackup and pack resulting directory via 7Zip, but the execution won't reach the second step. I can provide script source if it helps.
Any suggestions on how to investigate this? Which log files on Couchbase servers should I look into?
It seems my problem is related to this issue, but this one was fixed prior to 4.0 release.
I only now learned about -v flag on cbbackup and enabled it, will see what info it'll produce.
Since I started developing my application, I have used SQLite3. It has been working well, and I have filled my tables with data. Suddenly the terminal window which I used for querying the database stopped responding. I had an open SQLite3 terminal window when running a bundle install. Maybe that was causing the problems? I don't know, because I didn't try to use the terminal window until some hours later.
Now, when I run sqlite3 -line db/development.sqlite3, the usual things happen, but when I try to do a query or write ".tables", I get the error message shown in the title of this post. The strange thing is that querying the database from the application works well. Since the data obviously are intact, is there a way to back up these before I try to remove SQLite and install it again? How do I remove SQLite?
I tried bundle update SQLite3 and got no errors, but when I run sqlite3 -line db/development.sqlite3, the version reported is 3.7.7, and not 3.7.10 which is the newest version.
I use Ruby on Rails 3.0.10 on Ubuntu 11.10.