We are working with JBatch to execute a huge amount of jobs. We would like to disable the logging of the job execution to the JobRepository database in some circumstances, in best case on job level.
I haven't find any possibility to disable this logging in Glassfish /Payara Server, neither on job level nor entirely.
Did I miss anything or is this in general not possible?
Related
I am using hangfire as a custom workflow engine in a dotnet core application, which works perfectly fine until I start logging. I log to DB as a requirement. So normal database target. I enabled a sync=true. Once I turn logging on I start getting deadlocks. All is using DI. Again all works fine without log, once I add a rule to write the logs in db I get a deadlock. It’s not even a lot of concurrent jobs, only 11 in my test. Help pls?! Already lost a week on this and still stuck.
I am not sure it’s related, but I am also getting this at the same time: asynchronous exception: timeout flushing all targets
We have a DotNetNuke site running on two servers that are load balanced. To ensure the files are in sync on these servers, we are using File Replication Service.
Search works fine on DotNetNuke when not load balanced, but in the load balanced setup the search stops working after a while (no suggestions, no results).
The following related exception is all over our log files:
[D:2][T:31][ERROR] DotNetNuke.Services.Exceptions.Exceptions - Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: NativeFSLock#D:\Sites\SiteName\App_Data\Search\write.lock
at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit)
at Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, MaxFieldLength mfl)
at DotNetNuke.Services.Search.Internals.LuceneControllerImpl.get_Writer()
at DotNetNuke.Services.Search.Internals.LuceneControllerImpl.Delete(Query query)
at DotNetNuke.Services.Search.Internals.InternalSearchControllerImpl.DeleteSearchDocumentInternal(SearchDocument searchDocument, Boolean autoCommit)
at DotNetNuke.Services.Search.Internals.InternalSearchControllerImpl.DeleteSearchDocumentsByModule(Int32 portalId, Int32 moduleId, Int32 moduleDefId)
at DotNetNuke.Services.Search.SearchDataStore.StoreSearchItems(SearchItemInfoCollection searchItems)
at DotNetNuke.Services.Search.SearchEngine.IndexContent()
at DotNetNuke.Services.Search.SearchEngineScheduler.DoWork()
My best guess is that the issue is caused because both servers are running their search functionality, and the File Replication Service is syncing the files which causes conflicts.
What would be the best way to solve this?
Add an exclusion rule to not replicate the search index folder, but let both servers keep running search?
Somehow disable one server from indexing?
Any other suggestions?
Installation details:
DNN v. 09.02.00 (366)
.NET Framework 4.6
There's a 'scheduler' tool inside of the settings section that contains all CRON/background jobs functionality.
One of the background jobs is the 'Search: Site Crawler' job which is responsible for indexing the website. When that job runs at the same time on both servers, unexpected conflicts occur. To prevent this from happening, you can configure the job to only run on a specified server using the 'Servers' setting.
After configuring the job to only run on one server, the issue did not come back and search still works on both servers.
Thanks #Sanjay for pointing me in the right direction.
If I remember correctly search is done via a scheduled task. Have you tried setting up the task to run on only one server and then use file replication to sync across to the other server.
Cluster goes into deadlock state and stops allocating containers even when GBs of RAM and Vcores are available.
This was happening only when we start a lot of jobs in parallel most of which were Oozie jobs with many forked actions.
After a lot of search and reading related questions and articles, we came across a property called maxAMShare for YARN job scheduler (we are using Fair Scheduler).
What it means?
Percentage of memory and vcores from user's queue share that can be allotted to Application Masters. Default value: 0.5 (50%). Source
How it caused the deadlock?
When we will start multiple oozie jobs in parallel, each oozie job and the forked actions require couple of ApplicationMaster containers to be allotted first for oozie launchers which then start the other containers to do the actual action task.
In our case, we were actually starting around 20-30 oozie jobs in parallel, each with close to 20 forked actions. And with each action requiring 2 ApplicationMasters, close to 800 containers were getting blocked only by the Oozie ApplicationMasters.
Due to this, we were hitting the 50% default maxAMShare limit for our user queue. And YARN was not allowing to create new ApplicationMasters to run the actual job.
Solution?
One instant suggestion could be to disable the check by setting this property to -1.0. But this is not recommended. You can again end up allocating all or most of the resources to AMs and the real job that will get done will be very less.
Other option (which we went ahead with) is to specify a separate queue for AMs in the oozie configuration and then set maxAMShare property to 1.0. This way you can control how much resources can be allocated to AMs without affecting the other jobs. Reference
<global>
<configuration>
<property>
<name>oozie.launcher.mapred.job.queue.name</name>
<value>root.users.oozie_am_queue</value>
</property>
</configuration>
</global>
Hope this will be a major time saver for people facing the same issue. There could be many others reasons for deadlock too which are already discussed in other questions on SO.
I have an instance of APIConnect on premise.
Analyzing the logs, I have seen the task called "security-appID" moving from 10ms execution time to 200ms execution time.
What is the meaning of this task?
This task I believe offloads application security requests to other integrations if you have it so configured. It does not have anything to do necessarily with apiconnect, it is probably related to your bluemix ID, dashboard or landing page and how that is setup. You can probably find more information about it in the BMX docs. https://console.dys0.bluemix.net/docs/services/appid/existing.html#adding-app-id-to-an-existing-app
Rails 3 on Apache with Passenger: I need to take down Rails so that the DB connection(s) are closed -- this is necessary to perform some regular database maintenance -- but I want Apache up so that it can respond to requests with a static maintenance page.
I am using Capistrano and have seen the threads on how to invoke maintenance mode, but I need to know where to hook my DB tasks, and cannot figure out where.
Any links, or even pointers to where to look in the Capistrano code would be greatly appreciated.
TIA
You can use capistrano's deploy:web:disable task to block access to your site, allowing you to do database maintenance, etc:
cap deploy:web:disable REASON="a Database Upgrade" UNTIL="in a few minutes"
Then, once you're done:
cap deploy:web:enable