I've just created an extractor with import.io. This extractor uses chaining. Firstly I'm extracting some urls from one page and with these extracted urls, I'm extracting detail pages. When detail pages' extraction finish, I want to get the results. But how can I be sure that extraction is completed. Is there any api endpoint for checking the status of extraction?
I found "GET /store/connector/{id}" endpoint from legacy. But when I try this, I got 404. You can take a look at the screenshot.
Another question is, I want to schedule my extractor twice a day. Is this possible?
Thanks
Associated with each Extractor are Crawl Runs. A crawl run represents the running of an extractor with a specific configuration (training, list of URLs, etc). The state of each of a crawl run can have one of the following values:
STARTED => Currently running
CANCELLED => Started but cancelled by the user
FINISHED => Run was complete
Additional metadata that is included is as follows:
Started At - When the run started
Stopped At - When the run finished
Total URL Count - Total number of URLs in the run
Success URL Count - # of successful URLs queried
Failed URL Count - # of failed URLs queried
Row Count - Total number of rows returned in the run
The REST API to get the list of craw runs associated with an extractor is as follows:
curl -s X GET "https://store.import.io/store/crawlrun/_search?_sort=_meta.creationTimestamp&_page=1&_perPage=30&extractorId=$EXTRACTOR_ID&_apikey=$IMPORT_IO_API_KEY"
where
$EXTRACTOR_ID - Extractor to list crawl runs
$IMPORT_IO_API_KEY - Import.io API from your account
Related
I am trying to use Auth0 to get a list of users when my user list is >1000 (approx 2000)
So I understand a bit better now how this works after following the steps at:
https://auth0.com/docs/manage-users/user-migration/bulk-user-exports
There are three steps:
Use a POST call to the https://MY_DOMAIN/oauth/token endpoint to get an auth token (done)
Then take this token and insert it into the next POST call to the endpoint: https://MY_DOMAIN/api/v2/jobs/users-exports
Then take the job_id and insert it into the 3rd GET call to the endpoint: https://MY_DOMAIN/api/v2/jobs/MY_JOB_ID
But this just gives me a link to a document that I download. Essentially is the same end result as using the User Import / Export extension.
This is NOT what I want. I want to be able to call an endpoint and have it return a list of all the users (similar to the Retrieve Users with the Get Users Endpoint). I require it is done this way, so I can write a python script and run it as a cron job.
However, since I have over 1000 users, I am getting the below error when I call the GET /API/v2/users endpoint.
auth0.v3.exceptions.Auth0Error: 400: You can only page through the first 1000 records. See https://auth0.com/docs/users/search/v3/view-search-results-by-page#limitation
Can anyone help? Can this be done all the way I wish it to be?
I'm trying to get the job status by providing the build number as a parameter.
curl -s -S -u "Ashwin":"XXX" "http://XX.XXX.XXX.XX:8080/job/apitest/buildNum/logText/progressiveText?start=0"
The above snippet work absolutely fine. Is there anyway to send the build number as a json body.
In a word, no. The Jenkins API defines the query for the console log as a GET request, which (at least in the Jenkins API) does not contain a body. The primary parameters like job and build id are part of the URL path, and optional parameters are provided in the query string part of the URL.
This question strikes me as odd. Why can you not construct the query URL (which contains the build id as part of the URL path) in same way that you would construct the JSON structure that you propose sending in the body?
I run some load tests (all endpoints) and we do have a known issue in our code: if multiple POST requests are sent in the same time we do get a duplicate error based on a timestamp field in our database.
All I want to do is to count timeouts (based on the message received "Service is not available. Request timeout") in a variable and accept this as a normal behavior (don't fail the tests).
For now I've added a Response Assertion for this (in order to keep the tests running) but I cannot tell if or how many timeout actually happen.
How can I count this?
Thank you
I would recommend doing this as follows:
Add JSR223 Listener to your Test Plan
Put the following code into "Script" area:
if (prev.getResponseDataAsString().contains('Service is not available. Request timeout')) {
prev.setSampleLabel('False negative')
}
That's it, if sampler will contain Service is not available. Request timeout in the response body - JMeter will change its title to False negative.
You can even mark it as passed by adding prev.setSuccessful(false) line to your script. See Apache Groovy - Why and How You Should Use It article fore more information on what else you can do with Groovy in JMeter tests
If you just need to find out the count based on the response message then you can save the performance results in a csv file using simple data writer (configure for csv only) and then filter csv based on the response message to get the required count. Or you can use Display only "errors" option to get all the errors and then filter out based on the expected error message.
If you need to find out at the runtime then you can use aggregate report listener and use "Errors" checkbox to get the count of failure but this will include other failures also.
But, if you need to get the count at the run time to use it later then it is a different case. I am assuming that it is not the case.
Thanks,
What I need with the API call
I need the timeline info for each of the builds via the API call for Jenkins metrics plugin. I am using a web API to get data for my jobs in my jenkins. However, calling $JenkinsUrl/metrics/APIkey is leading me nowhere. Any idea how can I achieve the information?
All three bits of timeline info for each build (queue time, building time, total time) are available via the "get build" API.
On the screen where it shows "8.1 sec waiting in the queue" etc, click on the "REST API" link in the page footer, then "JSON API", then add &depth=2 to the end of the resulting API URL.
note, if you're searching for specific values in the API json, that the times will be in milliseconds. after you parse the json, build time is under the "duration" property, and the other two are in the array under the "actions" property. for me, it was the third element of the array, but that may vary (find the one with _class "jenkins.metrics.impl.TimeInQueueAction"):
{
"_class": "jenkins.metrics.impl.TimeInQueueAction",
"queuingDurationMillis": 16,
"totalDurationMillis": 4365
}
so in my example the build time was 4349 and the queue time was 16, so the total time was 4349 + 16 = 4365 milliseconds.
Instead of adding &depth=2 to the end of the url, you might be able to get exactly the three values you want by appending this to the url: &tree=duration,actions[queuingDurationMillis,totalDurationMillis]
I am new to jmeter and am working on putting together a test plan. The hurdle I've encountered is as follows:
First, a POST is made to processForm.aspx
Then, the user is redirected to pleaseWait.aspx
This page either redirects immediately to results.aspx OR loads, with a META REFRESH tag set to refresh in 5 seconds (and this step is repeated).
Now -- I can get this to execute by doing the following:
HTTP Sampler POST to processForm.aspx
Assert Response contains "<something on pleaseWait.aspx>"
While LAST
HTTP Sampler GET to pleaseWait.aspx
Assert Response contains "<something on results.aspx>"
However -- I don't care for this method, because it results in failed assertions (even though things are working as expected). I am sure there must be some other way to do this? Anyone more familiar with JMeter than I?
UPDATE:
Got it going by using Regular Expression Extractor instead of Assertions.
1) Add a User Defined Variables section at Test Plan Root
2) Add a variable to it "LoginWait" and "false"
HTTP Sampler POST to processForm.aspx
RegEx Extract Response Body contains "<something on pleaseWait.aspx>" into LoginWait
While ${LoginWait}
HTTP Sampler GET to pleaseWait.aspx
RegEx Extract Response Body contains "<something on pleaseWait.aspx>" into LoginWait
...
You could try using "follow redirects" on your HTTP Request. It would eliminate the logic you need, and still get you to the page you're going.