I have same code on two servers.
I have job for adding batch job to the queue
static void Job_ScheduleBatch2(Args _args)
{
BatchHeader batHeader;
BatchInfo batInfo;
RunBaseBatch rbbTask;
str sParmCaption = "My Demonstration (b351) 2014-22-09T04:09";
SysRecurrenceData sysRecurrenceData = SysRecurrence::defaultRecurrence();
;
sysRecurrenceData = SysRecurrence::setRecurrenceStartDateTime(sysRecurrenceData, DateTimeUtil::utcNow());
sysRecurrenceData = SysRecurrence::setRecurrenceUnit(sysRecurrenceData, SysRecurrenceUnit::Minute,1);
rbbTask = new Batch4DemoClass();
batInfo = rbbTask .batchInfo();
batInfo .parmCaption(sParmCaption);
batInfo .parmGroupId(""); // The "Empty batch group".
batHeader = BatchHeader ::construct();
batHeader .addTask(rbbTask);
batHeader.parmRecurrenceData(sysRecurrenceData);
//batHeader.parmAlerts(NoYes::Yes,NoYes::Yes,NoYes::Yes,NoYes::Yes,NoYes::Yes);
batHeader.addUserAlerts(curUserId(),NoYes::No,NoYes::No,NoYes::No,NoYes::Yes,NoYes::No);
batHeader .save();
info(strFmt("'%1' batch has been scheduled.", sParmCaption));
}
and I have a batch job
class Batch4DemoClass extends RunBaseBatch
{
int methodVariable1;
int metodVariable2;
#define.CurrentVersion(1)
#localmacro.CurrentList
methodVariable1,
metodVariable2
endmacro
}
public container pack()
{
return [#CurrentVersion,#CurrentList];
}
public void run()
{
// The purpose of your job.
info(strFmt("epeating batch job Hello from Batch4DemoClass .run at %1"
,DateTimeUtil ::toStr(
DateTimeUtil ::utcNow())
));
}
public boolean unpack(container _packedClass)
{
Version version = RunBase::getVersion(_packedClass);
switch (version)
{
case #CurrentVersion:
[version,#CurrentList] = _packedClass;
break;
default:
return false;
}
return true;
}
On one server it do runs and in batch history I can see it is saving messages to the batch log and on the other server it does nothing. One server is R2 (running) and R3 (not running).
Both codes are translated to CIL. Both servers are Batch server. I can see the batch jobs in USMF/System administration/Area page inquries. Only difference I can find is that jobs on R2 has filled AOS instance name in genereal and in R3 it is empty. In R2 I can see that info messages in log and batch history but there is nothing in R3.
Any idea how to make batch jobs running?
See how to Configure an AOS instance as a batch server.
There are 3 preconditions:
Marked as a batch server
Time interval set correctly
Batch group set correctly
Update:
Deleting the user of a batch job may make the batch job never complete. Once the execute queue has filled, no further progress will happen.
Deleting the offending batch jobs is problematic, as only the owner can do so! Then consider changing BatchJob.aosValidateDelete.
Yes, worth a try to clear the tables, also if you could provide a screenshot of the already running batchjobs and their status, that might be of help further.
Related
I am using Hangfire.AspNetCore 1.7.17 and Hangfire.MySqlStorage 2.0.3 for software that is currently in production.
Now and then, we get a report of jobs being executed twice, despite the usage of the [DisableConcurrentExecution] attribute with a timeout of 30 seconds.
It seems that as soon as those 30 seconds have passed, another worker picks up that same job again.
The code is fairly straightforward:
public async Task ProcessPicking(HttpRequest incomingRequest)
{
var filePath = await StoreStreamAsync(incomingRequest, TriggerTypes.Picking);
var picking = await XmlHelper.DeserializeFileAsync<Picking>(filePath);
// delay with 20 minutes so outbound-out gets the chance to be send first
BackgroundJob.Schedule(() => StartPicking(picking), TimeSpan.FromMinutes(20));
}
[TriggerAlarming("[IMPORTANT] Failed to parse picking message to **** object.")]
[DisableConcurrentExecution(30)]
public void StartPicking(Picking picking)
{
var orderlinePickModels = picking.ToSalesOrderlinePickQuantityRequests().ToList();
var orderlineStatusModels = orderlinePickModels.ToSalesOrderlineStatusRequests().ToList();
var isParsed = DateTime.TryParse(picking.Order.UnloadingDate, out var unloadingDate);
for (var i = 0; i < orderlinePickModels.Count; i++)
{
// prevents bugs with usage of i in the background jobs
var index = i;
var id = BackgroundJob.Enqueue(() => SendSalesOrderlinePickQuantityRequest(orderlinePickModels[index], picking.EdiReference));
BackgroundJob.ContinueJobWith(id, () => SendSalesOrderlineStatusRequest(
orderlineStatusModels.First(x=>x.SalesOrderlineId== orderlinePickModels[index].OrderlineId),
picking.EdiReference, picking.Order.PrimaryReference, isParsed ? unloadingDate : DateTime.MinValue));
}
}
[TriggerAlarming("[IMPORTANT] Failed to send order line pick quantity request to ****.")]
[AutomaticRetry(Attempts = 2)]
[DisableConcurrentExecution(30)]
public void SendSalesOrderlinePickQuantityRequest(SalesOrderlinePickQuantityRequest request, string ediReference)
{
var audit = new AuditPostModel
{
Description = $"Finished job to send order line pick quantity request for item {request.Itemcode}, part of ediReference {ediReference}.",
Object = request,
Type = AuditTypes.SalesOrderlinePickQuantity
};
try
{
_logger.LogInformation($"Started job to send order line pick quantity request for item {request.Itemcode}.");
var response = _service.SendSalesOrderLinePickQuantity(request).GetAwaiter().GetResult();
audit.StatusCode = (int)response.StatusCode;
if (!response.IsSuccessStatusCode) throw new TriggerRequestFailedException();
audit.IsSuccessful = true;
_logger.LogInformation("Successfully posted sales order line pick quantity request to ***** endpoint.");
}
finally
{
Audit(audit);
}
}
It schedules the main task (StartPicking) that creates the objects required for the two subtasks:
Send picking details to customer
Send statusupdate to customer
The first job is duplicated. Perhaps the second job as well, but this is not important enough to care about as it just concerns a statusupdate. However, the first job causes the customer to think that more items have been picked than in reality.
I would assume that Hangfire updates the state of a job to e.g. in progress, and checks this state before starting a job. Is my time-out on the disabled concurrent execution too low? Is it possible in this scenario that the database connection to update the state takes about 30 seconds (to be fair, it is running on a slow server with ~8GB Ram, 6 vCores) due to which the second worker is already picking up the job again?
Or is this a Hangfire specific issue that must be tackled?
I am new in ADF (EJB/JPA not Business Component), when the user is using our new app developed on jdeveloper "12.2.1.2.0", after an hour of activity, system is loosing the current record. To note that the object lost is the parent object.
I tried to change the session-timeout (knowing that it will affect the inactivity time).
public List<SelectItem> getSProvMasterSelectItemList(){
List<SelectItem> sProvMasterSelectItemList = new ArrayList<SelectItem>();
DCIteratorBinding lBinding = ADFUtils.findIterator("pByIdIterator");/*After 1 hour I am able to get lBinding is not null*/
Row pRow = lBinding.getCurrentRow();/*But lBinding.getCurrentRow() is null*/
DCDataRow objRow = (DCDataRow) pRow;
Prov prov = (Prov) objRow.getDataProvider();
if (!StringUtils.isEmpty(prov)){
String code = prov.getCode();
if (StringUtils.isEmpty(code)){
return sProvMasterSelectItemList;
}else{
List<Lov> mProvList = getSessionEJBBean().getProvFindMasterProv(code);
sProvMasterSelectItemList.add(new SelectItem(null," "));
for (Lov pMaster:mProvList) {
sProvMasterSelectItemList.add(new SelectItem(pMaster.getId(),pMaster.getDescription()));
}
}
}
return sProvMasterSelectItemList ;
}
I expect to be able to read the current record at any time, specially that it is the master block, and one record is available.
This look like a classic issue of misconfigured Application Module.
Cause : Your application module is timing out and releasing it's transaction before the official adfc-config timeout value.
To Fix :
Go to the application module containing this VO > Configuration > Edit the default > Modify Idle Instance Timeout to be the same as your adf session timeout (Take time to validate the other configuration aswell)
template.setEnableTransactionSupport(true);
template.multi();
template.opsForValue().set("mykey", "Hello World");
List<String> dataList = template.opsForList().range("mylist", 0, -1);
template.exec();
Hi guys.
I have a list called "mylist' in my redis and its size is 50.
But when I run this code, I can't get what I want.
The field "dataList" is null, however, "mykey" with the value "Hello World" have persisted in my redis.
So how can I get my list data in a spring-data-redis transaction? many thanks.
The transaction suppport in SD-Redis helps participating in ongoing transactions and allows automatic commit (exec) / rollback (discard), so it helps wrapping commands into thread bound multi exec blocks using the same connection.
More generally redis transactions and the commands within a transaction get queued on server side and return a list of results on exec.
template.multi();
// queue set command
template.opsForValue().set("mykey", "Hello World");
// queue range command
List<String> dataList = template.opsForList().range("mylist", 0, -1);
// execute queued commands
// result[0] = OK
// result[1] = {"item-1", "item-2", "item-", ...}
List<Object> result = template.exec();
1) I setup a private ethereum network using the following command
$geth --genesis <genesis json file path> --datadir <some path to an empty
folder> --networkid 123 --nodiscover --maxpeers 0 console
2) Created an account
3) Then, started the miner using miner.start() command.
After a while ethers were getting added automatically to my account, but I don’t have any pending transaction in my private network. So from where could my miners are getting the ethers?
Even though, I didn’t instantiate any transactions in my network, I could see the some transaction being recorded in the logs, once I start the miner.
The log is as follows:
I0118 11:59:11.696523 9427 backend.go:584] Automatic pregeneration of ethash
DAG ON (ethash dir: /Users/minisha/.ethash)
I0118 11:59:11.696590 9427 backend.go:591] checking DAG (ethash dir:
/Users/minisha/.ethash)
I0118 11:59:11.696728 9427 miner.go:119] Starting mining operation (CPU=4
TOT=5)
true
> I0118 11:59:11.703907 9427 worker.go:570] commit new work on block 1 with 0
txs & 0 uncles. Took 7.109111ms
I0118 11:59:11.704083 9427 ethash.go:220] Generating DAG for epoch 0 (size
1073739904) (0000000000000000000000000000000000000000000000000000000000000000)
I0118 11:59:12.698679 9427 ethash.go:237] Done generating DAG for epoch 0, it
took 994.61107ms
I0118 11:59:15.163864 9427 worker.go:349]
And my genesis block code is as follows:
{
“nonce”: “0xdeadbeefdeadbeef”,
“timestamp”: “0x0”,
“parentHash”:
“0x0000000000000000000000000000000000000000000000000000000000000000”,
“extraData”: “0x0”,
“gasLimit”: “0x8000000”,
“difficulty”: “0x400”,
“mixhash”:
“0x0000000000000000000000000000000000000000000000000000000000000000”,
“coinbase”: “0x3333333333333333333333333333333333333333”,
“alloc”: {
}
}
Since my network is isolated and have only one node (no peers), I am quite confused with this behaviour. Any insights would be greatly appreciated.
Your client is mining empty blocks (containing no transactions) and getting rewards for mined blocks what is 5 ETH per block.
If you want to prevent empty block in your private blockchain you should consider using eth client (C++ implementation).
In case of geth client you can use a JavaScript script that will modify client's behavior. Any script can be loaded with js command: geth js script.js.
var mining_threads = 1
function checkWork() {
if (eth.getBlock("pending").transactions.length > 0) {
if (eth.mining) return;
console.log("== Pending transactions! Mining...");
miner.start(mining_threads);
} else {
miner.stop(0); // This param means nothing
console.log("== No transactions! Mining stopped.");
}
}
eth.filter("latest", function(err, block) { checkWork(); });
eth.filter("pending", function(err, block) { checkWork(); });
checkWork();
You can try EmbarkJS which can run geth client with mineWhenNeeded option on private network. It will only mine when new transactions come in.
I am uploading data to big query as csv format with JSON schemas. What I am seeing is the very long times to load into big query. I take the start and ending load times from the pollJob.getStatistics() when the load is DONE and compute a delta time as (startTime - endTime)/1000. Then I look at the number of bytes loaded. The data is from files stored in google cloud storage that I reprocess in app engine to do some reformatting. I convert the string into a byte stream and then load as the contents of the load as follows:
public static void uploadFileToBigQuerry(TableSchema tableSchema,String tableData,String datasetId,String tableId,boolean formatIsJson,int waitSeconds,String[] fileIdElements) {
/* Init diagnostic */
String projectId = getProjectId();
if (ReadAndroidRawFile.testMode) {
String s = String.format("My project ID at start of upload to BQ:%s datasetID:%s tableID:%s json:%b \nschema:%s tableData:\n%s\n",
projectId,datasetId,tableId,formatIsJson,tableSchema.toString(),tableData);
log.info(s);
}
else {
String s = String.format("Upload to BQ tableID:%s tableFirst60Char:%s\n",
tableId,tableData.substring(0,60));
log.info(s);
}
/* Setup the data each time */
Dataset dataset = new Dataset();
DatasetReference datasetRef = new DatasetReference();
datasetRef.setProjectId(projectId);
datasetRef.setDatasetId(datasetId);
dataset.setDatasetReference(datasetRef);
try {
bigquery.datasets().insert(projectId, dataset).execute();
} catch (IOException e) {
if (ReadAndroidRawFile.testMode) {
String se = String.format("Exception creating datasetId:%s",e);
log.info(se);
}
}
/* Set destination table */
TableReference destinationTable = new TableReference();
destinationTable.setProjectId(projectId);
destinationTable.setDatasetId(datasetId);
destinationTable.setTableId(tableId);
/* Common setup line */
JobConfigurationLoad jobLoad = new JobConfigurationLoad();
/* Handle input format */
if (formatIsJson) {
jobLoad.setSchema(tableSchema);
jobLoad.setSourceFormat("NEWLINE_DELIMITED_JSON");
jobLoad.setDestinationTable(destinationTable);
jobLoad.setCreateDisposition("CREATE_IF_NEEDED");
jobLoad.setWriteDisposition("WRITE_APPEND");
jobLoad.set("Content-Type", "application/octet-stream");
}
else {
jobLoad.setSchema(tableSchema);
jobLoad.setSourceFormat("CSV");
jobLoad.setDestinationTable(destinationTable);
jobLoad.setCreateDisposition("CREATE_IF_NEEDED");
jobLoad.setWriteDisposition("WRITE_APPEND");
jobLoad.set("Content-Type", "application/octet-stream");
}
/* Setup the job config */
JobConfiguration jobConfig = new JobConfiguration();
jobConfig.setLoad(jobLoad);
JobReference jobRef = new JobReference();
jobRef.setProjectId(projectId);
Job outputJob = new Job();
outputJob.setConfiguration(jobConfig);
outputJob.setJobReference(jobRef);
/* Convert input string into byte stream */
ByteArrayContent contents = new ByteArrayContent("application/octet-stream",tableData.getBytes());
int timesToSleep = 0;
try {
Job job = bigquery.jobs().insert(projectId,outputJob,contents).execute();
if (job == null) {
log.info("Job is null...");
throw new Exception("Job is null");
}
String jobIdNew = job.getId();
//log.info("Job is NOT null...id:");
//s = String.format("job ID:%s jobRefId:%s",jobIdNew,job.getJobReference());
//log.info(s);
while (true) {
try{
Job pollJob = bigquery.jobs().get(jobRef.getProjectId(), job.getJobReference().getJobId()).execute();
String status = pollJob.getStatus().getState();
String errors = "";
String workingDataString = "";
if ((timesToSleep % 10) == 0) {
String statusString = String.format("Job status (%dsec) JobId:%s status:%s\n", timesToSleep, job.getJobReference().getJobId(), status);
log.info(statusString);
}
if (pollJob.getStatus().getState().equals("DONE")) {
status = String.format("Job done, processed %s bytes\n", pollJob.getStatistics().toString()); // getTotalBytesProcessed());
log.info(status); // compute load stats with this string
if ((pollJob.getStatus().getErrors() != null)) {
errors = pollJob.getStatus().getErrors(). toString();
log.info(errors);
}
The performance I get is as follows: the median upload of BYTES/(deltaTime) is 17 BYTES/sec! Yes, bytes, not kilo or mega...
Worse is that sometimes for only a few hundred bytes, just one row, it takes up to 5 minutes. I generally have no errors, but I am thinking that with this performance, I will not be able to upload each app before more data arrives. I am processing with a task queue in a backends instance. This task queue gets a time-out after about an hour of processing.
Is this poor performance because of the contents method?
A couple of things:
If you are loading a small amount of data, you may be better off using TableData.insertAll() rather than a load job, which lets you post the data and have it be available immediately.
Load jobs are Batch oriented jobs. That is, you can insert (more or less) as many as you'd like and they'll be processed when there are resources to do so. Sometimes you create a job and the worker pool is resizing so you have to wait. Sometimes the worker pool is full.
If you provide a project & Job ID we can look into the performance of individual jobs to see what's taking so long.
Load jobs process in parallel; that is, once they start executing they should go very quickly, but the time to start executing may take a long time.
There are three time fields in the job statistics. createTime, startTime, and endTime.
createTime is the moment the BigQuery server receives your request.
startTime is when BigQuery actually starts working on your job
endTime is when the job is completely done
I'd expect that most of the time is being spent between create and start. If that is not the case for small jobs, then it means that something is strange is going on, and a Job ID would help diagnose the issue.