newbie in apache nutch - writing a client to use it via REST.
succeed in all the steps (INJECT, FETCH...) - in the last step - when trying to index to solr - it fails to pass the parameter.
The Request (I formatted it in some website)
{
"args": {
"batch": "1463743197862",
"crawlId": "sample-crawl-01",
"solr.server.url": "http:\/\/x.x.x.x:8081\/solr\/"
},
"confId": "default",
"type": "INDEX",
"crawlId": "sample-crawl-01"
}
The Nutch logs:
java.lang.Exception: java.lang.RuntimeException: Missing SOLR URL. Should be set via -D solr.server.url
SOLRIndexWriter
solr.server.url : URL of the SOLR instance (mandatory)
solr.commit.size : buffer size when sending to SOLR (default 1000)
solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
solr.auth : use authentication (default false)
solr.auth.username : username for authentication
solr.auth.password : password for authentication
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Was that implemented? the param passing to solr plugin?
You need to create/update a configuration using the /config/create/ endpoint, with a POST request and a payload similar to:
{
"configId":"solr-config",
"force":"true",
"params":{"solr.server.url":"http://127.0.0.1:8983/solr/"}
}
In this case I'm creating a new configuration and specifying the solr.server.url parameter. You can verify this is working with a GET request to /config/solr-config (solr-config is the previously specified configId), the output should contain all the default parameters see https://gist.github.com/jorgelbg/689b1d66d116fa55a1ee14d7193d71b4 for an example/default output. If everything worked fine in the returned JSON you should see the solr.server.url option with the desired value https://gist.github.com/jorgelbg/689b1d66d116fa55a1ee14d7193d71b4#file-nutch-solr-config-json-L464.
After this just hit the /job/create endpoint to create a new INDEX Job, the payload should be something like:
{
"type":"INDEX",
"confId":"solr-config",
"crawlId":"crawl01",
"args": {}
}
The idea is that need to you pass the configId that you created with the solr.server.url specified along with the crawlId and other args. This should return something similar to:
{
"id": "crawl01-solr-config-INDEX-1252914231",
"type": "INDEX",
"confId": "solr-config",
"args": {},
"result": null,
"state": "RUNNING",
"msg": "OK",
"crawlId": "crawl01"
}
Bottom line you need to create a new configuration with the solr.server.url setted instead of specifying it through the args key in the JSON payload.
Related
I've been trying to set up a new custom webhook in Zapier, that automatically updates a custom field in JIRA, whenever a specific action occurs. I've followed some tutorials on how to do it, but when I sent the PUT request, it didn't work. I also tested a bunch in postman, but with similar results.
I used this URL:
https://bitsandbirds.atlassian.net/rest/api/3/issue/CYBIRD-1252
Here is my input:
{
"update" : {
"customfield_10051" : "test"
}
}
This is what I got back:
{
"errorMessages": [
"Can not deserialize instance of java.util.ArrayList out of VALUE_STRING token\n at [Source: org.apache.catalina.connector.CoyoteInputStream#498ac517; line: 3, column: 8] (through reference chain: com.atlassian.jira.rest.v2.issue.IssueUpdateBean[\"update\"])"
]
}
Anyone know where I messed up & how to do it right?
fyi here is the view in Zapier
There should be
{
"fields": {
"customfield_10051" : "test"
}
}
I am trying to create a controller service using nifi api rest but I am blocked because when I try:
InvokeHTTP
POST
https://hostname/nifi-api/controller/controller-services
using this json
{
"revision": {
"version": 0
},
"disconnectedNodeAcknowledged": false,
"component": {
"name": "DMCS_try",
"type": "org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer",
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-distributed-cache-services-nar",
"version": "1.9.0.3.4.1.1-4"
},
"state": "ENABLED",
"properties": {
"Port": "4555",
"Maximum Cache Entries": "10000",
"Eviction Strategy": null,
"Persistence Directory": null,
"SSL Context Service": null
}
}
}
I got this "error"
Node XXXXXXXXX is unable to fulfill this request due to: Unable to modify the controller. Contact the system administrator. Contact the system administrator.
Controller services can be created in two different places. One place is in the flow as part of a process group so they can be used by processors, the other place is at the controller level for use by reporting tasks.
The URL you specified is for creating a service at the controller level and therefore whatever identity you are using to authenticate as need permissions to modify the controller (WRITE on /controller). The error message is saying you don't have that permission.
I have some API like https://api-******y?apikey=sX4&q=london, Where q=london is a parameter that will keep changing, So how can i add this URL in the list of urls in ngsw-config.json file
"dataGroups": [
{
"name": "api-performance",
"urls": [
"https://api-***************y?apikey=sX******4&q=london",
"https://api-***************y?apikey=sX******4&q=manchester",
"https://api-***************y?apikey=sX******4&q=birmingham"
],
"cacheConfig": {
"strategy": "performance",
"maxSize": 100,
"maxAge": "3d"
}
},
I have two doubts:
Instead of adding same url again and again for all different parameters like london/manchester/birmingham is there a way that i can just add the base URL without parameter and it will store the responses for all different parameters when user will search for it?
Instead of adding the URL's manually, the URL's response should be stored dynamically when the user has searched for it when he was online. I mean i will initially not add any URL in list in the ngsw-config.json file, BUT when user has requested for that url, its response automatically get stored in cache.
I got this one, Its actually a little stupid of me to think that i have to give the complete url in the ngsw-config.json file,
Actually we just can simply give the base url, we neither have to give the apiKey parameter nor any other parameter like london/manchester/birmingham, ngsw.json file in dist folder after the build, stores the url as patterns, it just matches the base url and stores all the queries with all different parameters made to that url by user. above code can re-written like below. it will work fine by caching the data for all the stores london/manchester/birmingham, whenever user searches for them.
"dataGroups": [
{
"name": "api-performance",
"urls": [
"https://api-***************y
],
"cacheConfig": {
"strategy": "performance",
"maxSize": 100,
"maxAge": "3d"
}
},
I need to call a Jenkins job using its API through Postman. This job requires parameters (HOST, VERBOSITY and PMSP).
Auth works using Jenkins token and header Content-type:application/json is used.
I tried to call the endpoint https://jenkins_server/job/job_name/build/api/json adding the following body to the request but the result is Nothing is submitted, and the job doesn't run.
I tried to call the endpoint https://jenkins_server/job/job_name/buildWithParameters/api/json adding the same body. I get 201 Created (job is running) but no parameters are given to the job.
{
"parameter": [
{
"name": "HOSTS",
"value": "[linux]\n1.2.3.4"
},
{
"name": "VERBOSITY",
"value": "vv"
},
{
"name": "SANS_PMSP",
"value": true
}
]
}
Is my JSON well constructed ? Which endpoint do I need to call ?
If it's Postman that you would like to focus on, you can import the curl command straight into the application.
This creates a new request for you to use and it populates this request, based on the details in the command.
From here, you should be able to add your own URL and point this at the location you need.
I am using IBM Worklight Studio, and trying to create HTTP Adapter which retrieve JSON object from external http server.
When I just access target http server with HTTP Get access(with browser, for example), I know their response is like following array style JSON format:
[
{ "xxx":"aaa", "yyy":"bbb", ... },
{ "xxx":"ccc", "yyy":"ddd", ... },
:
{ "xxx":"eee", "yyy":"fff", ... }
]
And I had created HTTP Adapter which would retrieve above information
var input = {
method : 'get',
returnedContentType : 'json',
path : path
};
return WL.Server.invokeHttp(input);
Now I tried to invoke this adapter with "Run As -> Invoke Worklight Procedure", then I got this error message:
{
"errors": [
"Runtime: Failed to parse JSON string\n\n[\n {\n
(raw JSON data) } ],
"info": [],
"isSuccessful": false,
"warnings": []
}
And in my log console, worklight says following error messages:
FWLSE0101E: Caused by: java.io.IOException: Expecting '{' on line 2, column 2 instead, obtained token: 'Token: ['
From above information, it seems that worklight would expect that returned JSON object need to start with "{", not "[".
Is this my guess right? Are there any workaround for this?
Thanks for advance.
Worklight knows how to handle JSON objects that start with [ (JSON arrays). In such case Worklight will return the response as:
{ "array" : [*the json array*]}
Looking at the code of the HTTP Adapter, I see that there is a bug with parsing JSON arrays that do not start with [.
I do not see a workaround for this problem, except changing the response returned from the http server.
I opened an internal bug about this, thank you for helping us find this bug.
You can change returnedContentType to "plain", this will make WL server return content as a big string and not attempt to parse it.
Then in your adapter you can use var obj = JSON.parse(response.text)