I have this test
defmodule InfoSys.Backends.WolframTest do
use ExUnit.Case, async: true
alias InfoSys.WolFram
test "make request, report results, then terminates" do
ref = make_ref()
{:ok, pid} = WolFram.start_link("1 + 1", ref, self(), 1)
assert_receive {:results, ^ref, [%InfoSys.Result{text: "2"}]}
end
end
and I am receiving
No message matching {:results, ^ref, [%InfoSys.Result{text: "2"}]}
after 100ms.
How can I know which message is ref receiving or how can I debug this error? I am following phoenix book programming example
Whole ExUnit stacktrace:
1) test make request, report results, then terminates (InfoSys.Backends.WolframTest)
test/backends/wolfram_test.exs:6
No message matching {:results, ^ref, [%InfoSys.Result{text: "2"}]} after 100ms.
The following variables were pinned:
ref = #Reference<0.214527998.4009754625.219099>
The process mailbox is empty.
code: assert_receive {:results, ^ref, [%InfoSys.Result{text: "2"}]}
stacktrace:
test/backends/wolfram_test.exs:11: (test)
I've ran on these kind of problems before. There are two probable causes:
Your process is taking longer than 100 ms to give you the result.
Your process has a bug.
I prefer to rule out the first possibility and the thing I do is making the test wait more than the default time which is 100 ms. assert_receive can receive a second argument which is the time in milliseconds. You can try something like this:
assert_receive {:results, ^ref, [%InfoSys.Result{text: "2"}]}, 5000
where 5000 means 5000 ms or 5 seconds.
After increasing the time, there are two possible outcomes when running a test like this:
Your test mailbox is not empty: your process is slow when processing your requests. Nothing wrong with the logic. Maybe the thing your process is doing is an expensive operation or you need to improve your code's efficiency.
Your test mailbox is empty:
There is definitely a problem with the logic of your process and you
should check the code.
You have underestimated the complexity of the operation your process should do and you should find a more efficient way of doing it or change your solution.
I've seen the code you are taking about and it's not too complex (next time you should post it as part of the question), so if you don't receive a response in your test after 5 seconds, then there is definitely something wrong with your process' code.
I hope this helps.
Related
I´ve got a Job that runs a TaskletStep, then a chunk-based step and then another TaskletStep.
In each of these steps, errors (in the form of Exceptions) can occur.
The chunk-based step looks like this:
stepBuilderFactory
.get("step2")
.chunk<SomeItem, SomeItem>(1)
.reader(flatFileItemReader)
.processor(itemProcessor)
.writer {}
.faultTolerant()
.skipPolicy { _ , _ -> true } // skip all Exceptions and continue
.taskExecutor(taskExecutor)
.throttleLimit(taskExecutor.corePoolSize)
.build()
The whole job definition:
jobBuilderFactory.get("job1")
.validator(validator())
.preventRestart()
.start(taskletStep1)
.next(step2)
.next(taskletStep2)
.build()
I expected that Spring Batch somehow picks up the Exceptions that occur along the way, so I can then create a Report including them after the Job has finished processing. Looking at the different contexts, there´s also fields that should contain failureExceptions. However, it seems there´s no such information (especially for the chunked step).
What would be a good approach if I need information about:
what Exceptions did occur in which Job execution
which Item was the one that triggered it
The JobExecution provides a method to get all failure exceptions that happened during the job. You can use that in a JobExecutionListener#afterJob(JobExecution jobExecution) to generate your report.
In regards to which items caused the issue, this will depend on where the exception happens (during the read, process or write operation). For this requirement, you can use one of the ItemReadListener, ItemProcessListener or ItemWriteListener to keep record of the those items (For example, by adding them to the job execution context to be able to get access to them in the JobExecutionListener#afterJob method for your report).
I am using the taskqueue API to send multiple emails is small groups with mailgun. My code looks more or less like this:
class CpMsg(ndb.Model):
group = ndb.KeyProperty()
sent = ndb.BooleanProperty()
#Other properties
def send_mail(messages):
"""Sends a request to mailgun's API"""
# Some code
pass
class MailTask(TaskHandler):
def post(self):
p_key = utils.key_from_string(self.request.get('p'))
msgs = CpMsg.query(
CpMsg.group==p_key,
CpMsg.sent==False).fetch(BATCH_SIZE)
if msgs:
send_mail(msgs)
for msg in msgs:
msg.sent = True
ndb.put_multi(msgs)
#Call the task again in COOLDOWN seconds
The code above has been working fine, but according to the docs, the taskqueue API guarantees that a task is delivered at least once, so tasks should be idempotent. Now, most of the time this would be the case with the above code, since it only gets messages that have the 'sent' property equal to False. The problem is that non ancestor ndb queries are only eventually consistent, which means that if the task is executed twice in quick succession the query may return stale results and include the messages that were just sent.
I thought of including an ancestor for the messages, but since the sent emails will be in the thousands I'm worried that may mean having large entity groups, which have a limited write throughput.
Should I use an ancestor to make the queries? Or maybe there is a way to configure mailgun to avoid sending the same email twice? Should I just accept the risk that in some rare cases a few emails may be sent more than once?
One possible approach to avoid the eventual consistency hurdle is to make the query a keys_only one, then iterate through the message keys to get the actual messages by key lookup (strong consistency), check if msg.sent is True and skip sending those messages in such case. Something along these lines:
msg_keys = CpMsg.query(
CpMsg.group==p_key,
CpMsg.sent==False).fetch(BATCH_SIZE, keys_only=True)
if not msg_keys:
return
msgs = ndb.get_multi(msg_keys)
msgs_to_send = []
for msg in msgs:
if not msg.sent:
msgs_to_send.append(msg)
if msgs_to_send:
send_mail(msgs_to_send)
for msg in msgs_to_send:
msg.sent = True
ndb.put_multi(msgs_to_send)
You'd also have to make your post call transactional (with the #ndb.transactional() decorator).
This should address the duplicates caused by the query eventual consistency. However there still is room for duplicates caused by transaction retries due to datastore contention (or any other reason) - as the send_mail() call isn't idempotent. Sending one message at a time (maybe using the task queue) could reduce the chance of that happening. See also GAE/P: Transaction safety with API calls
I am using parallel processing.
CALL FUNCTION 'ZABC' STARTING NEW TASK taskname
DESTINATION IN GROUP srv_grp PERFORMING come_back ON END OF TASK
EXPORTING
...
EXCEPTIONS
...
.
I am calling this FM inside a loop. sometimes, my records are skipped. I am not getting a desired output. Sometimes 2000 records are processed and sometimes 1000. The numbers are varying. what can be the problem ? Can you provide me some cases where records can be skipped in parallel processing ?
See the example code SAP provides: https://help.sap.com/viewer/753088fc00704d0a80e7fbd6803c8adb/7.4.16/en-US/4892082ffeb35ed2e10000000a42189d.html
Look at how they handle the resource exceptions. A resource exception means that you tried to start a new aRFC but there were no more processes available. Your program has to handle these cases. Without handling these cases the entries will be skipped. The normal handling is to just wait a little time for some of the active processes to finish, in the example program:
WAIT UNTIL rcv_jobs >= snd_jobs
UP TO 5 SECONDS.
gv_semaphore = 0.
" your internal table size.
DESCRIBE TABLE lt_itab LINES lv_lines.
LOOP AT lt_itab INTO ls_itab.
" parallel proc. -->>
CALL FUNCTION 'ZABC' STARTING NEW TASK taskname
DESTINATION IN GROUP srv_grp PERFORMING come_back ON END OF TASK
EXPORTING
...
EXCEPTIONS
...
.
" <<--
ENDLOOP.
" wait until all parallel processes have terminated. **
WAIT UNTIL gv_semaphore = lv_lines.
NOTE: You should check total parallel process count. There should be some upper limits for opened threads.
Thanks.
I am facing the following problem:
I have multiple HTTP Requests in my testplan.
I want every request to be repeated 4 times if they fail.
I realized that with a BeanShell Assertion, and its already working fine.
My problem is, that I don't want requests to be executed if a previous Request failed 5 times,
BUT I also dont want the thread to end.
I just want the current thread iteration to end,
so that the next iteration of the thread can start again with the 1st request (if the thread is meant to be repeated).
How do I realize that within the BeanShell Assertion?
Here is just a short extract of my code where i want the solution to have
badResponseCounter is being increased for every failed try of the request, this seems to work so far. Afterwards, the variable gets resetted.
if (badResponseCounter = 5) {
badResponseCounter = 0;
// Stop current iteration
}
I already checked the API, methods like setStopTest() or setStopThread() are given, but nothing for quitting the current iteration. I also need the preference "continue" in the thread group, as otherwise the entire test will stop after 1 single request failed.
Any ideas of how to do this?
In my opinion the easiest way is using following combination:
If Controller to check ${JMeterThread.last_sample_ok} and badResponseCounter variables
Test Action Sampler as a child of If Controller configured to "Go to next loop iteration"
Try this.
ctx.setRestartNextLoop(true);
if the thread number is 2, i tried to skip. I get the below result as i expected (it does not call b-2). It does not kill the thread either.
We run multiple short queries in parallel, and hit the 10 sec limit.
According to the docs, throttling might occur if we hit a limit of 10 API requests per user per project.
We send a "start query job", and then we call the "getGueryResutls()" with timeoutMs of 60,000, however, we get a response after ~ 1 sec, we look for JOB Complete in the JSON response, and since it is not there, we need to send the GetQueryResults() again many times and hit the threshold, that is causing an error, not a slowdown. the sample code is below.
our questions are as such:
1. What is a "user" is it an appengine user, is it a user-id that we can put in the connection string or in the query itslef?
2. Is it really per API project of BigQuery?
3. What is the behavior?we got an error: "Exceeded rate limits: too many user/method api request limit for this user_method", and not a throttling behavior as the doc say and all of our process fails.
4. As seen below in the code, why we get the response after 1 sec & not according to our timeout? are we doing something wrong?
Thanks a lot
Here is the a sample code:
while (res is None or 'jobComplete' not in res or not res['jobComplete']) :
try:
res = self.service.jobs().getQueryResults(projectId=self.project_id,
jobId=jobId, timeoutMs=60000, maxResults=maxResults).execute()
except HTTPException:
if independent:
raise
Are you saying that even though you specify timeoutMs=60000, it is returning within 1 second but the job is not yet complete? If so, this is a bug.
The quota limits for getQueryResults are actually currently much higher than 10 requests per second. The reason the docs say only 10 is because we want to have the ability to throttle it down to that amount if someone is hitting us too hard. If you're currently seeing an error on this API, it is likely that you're calling it at a very high rate.
I'll try to reproduce the problem where we don't wait for the timeout ... if that is really what is happening it may be the root of your problems.
def query_results_long(self, jobId, maxResults, res=None):
start_time = query_time = None
while res is None or 'jobComplete' not in res or not res['jobComplete']:
if start_time:
logging.info('requested for query results ended after %s', query_time)
time.sleep(2)
start_time = datetime.now()
res = self.service.jobs().getQueryResults(projectId=self.project_id,
jobId=jobId, timeoutMs=60000, maxResults=maxResults).execute()
query_time = datetime.now() - start_time
return res
then in appengine log I had this:
requested for query results ended after 0:00:04.959110