Couchdb bad_utf8_character_code - ruby-on-rails-3

I am using couchdb through couchrest in a ruby on rails application. When I try to use futon it alerts with a message box saying bad_utf8_character_code. If I try to access records from rails console using Model.all it raises either of 500:internal server error,
RestClient::ServerBrokeConnection: Server broke connection: or Errno::ECONNREFUSED: Connection refused - connect(2)
Could any one help me to sort this issue?

I ran into this issue. I tried various curl calls to delete, modify, and even just view the offending document. None of them worked. Finally, I decided to pull the documents down to my local machine one at a time, skip the "bad" one, and then replicate from my local out to production.
Disable app (so no more records are being written to the db)
Delete and recreate local database (run these commands in a shell):
curl -X DELETE http://127.0.0.1:5984/mydb
curl -X PUT http://127.0.0.1:5984/mydb
Pull down documents from live to local using this ruby script
require 'bundler'
require 'json'
all_docs = JSON.parse(`curl http://server.com:5984/mydb/_all_docs`)
docs = all_docs['rows']
ids = docs.map{|doc| doc['id']}
bad_ids = ['196ee4a2649b966b13c97672e8863c49']
good_ids = ids - bad_ids
good_ids.each do |curr_id|
curr_doc = JSON.parse(`curl http://server.com:5984/mydb/#{curr_id}`)
curr_doc.delete('_id')
curr_doc.delete('_rev')
data = curr_doc.to_json.gsub("\\'", "\'").gsub('"','\"')
cmd = %Q~curl -X PUT http://127.0.0.1:5984/mydb/#{curr_id} -d "#{data}"~
puts cmd
`#{cmd}`
end
Destroy (delete) and recreate production database (I did this in futon)
Replicate
curl -X POST http://127.0.0.1:5984/_replicate -d '{"target":"http://server.com:5984/mydb", "source":"http://127.0.0.1:5984/mydb"}' -H "Content-Type: application/json"
Restart app

Related

import status lingers after GraphDB repository deleted

GraphDB Free/9.4.1, RDF4J/3.3.1
I'm working on using the /rest/data/import/server/{repo-id} endpoint to initiate the importing of an RDF/XML file.
Steps:
put SysML.owl in the ${graphdb.workbench.importDirectory} directory.
chmod a+r SysML.owl
create repository test1 (in Workbench - using all defaults except RepositoryID := "test1")
curl http://127.0.0.1:7200/rest/data/import/server/test1 => as expected:
[{"name":"SysML.owl","status":"NONE"..."timestamp":1606848520821,...]
curl -XPOST --header 'Content-Type: application/json' --header 'Accept: application/json' -d ' { "fileNames":[ "SysML.owl" ] }' http://127.0.0.1:7200/rest/data/import/server/test1 => SC==202
after 60 seconds, curl http://127.0.0.1:7200/rest/data/import/server/test1 =>
[{"name":"SysML.owl","status":"DONE","message":"Imported successfully in 7s.","context":null,"replaceGraphs":[],"baseURI":
"file:/home/steve/graphdb-import/SysML.owl", "forceSerial":false,"type":"file","format":null,"data":null,"timestamp":
1606848637716, [...other json content deleted]
Repository test1 now has the 263,119 (824 inferred) statements from SysML.owl loaded
BUT if I then
delete the repository using the Workbench page at http://localhost:7200/repository, wait 180 seconds
curl http://127.0.0.1:7200/rest/data/import/server/test => same as in step 5 above, despite repository having been deleted.
curl -X GET --header 'Accept: application/json' 'http://localhost:7200/rest/repositories' => test1 not shown.
create the repository again, using the Workbench - same settings as previously. wait 60 seconds. Initial 70 statements present.
curl http://127.0.0.1:7200/rest/data/import/server/test1 =>
The same output as from the earlier usage - when I was using the prior repository instance. "status":"DONE", same timestamp - which is prior to the time at which I deleted, recreated the test1 repository.
The main-2020-12-01.log shows the INFO messages pertaining to the repository test1, plugin registrations, etc. Nothing indicating why the prior repository instance's import status is lingering.
And this is of concern because I was expecting to use some polling of the status to determine when the data is loaded so my processing can proceed. Some good news - I can issue the import server file request again and after waiting 60 seconds, the 263,119 statements are present. But the timestamp on the import is the earlier repo instance's timestamp. It was not reset via the latest import request.
I'm probably missing some cleanup step(s), am hoping someone knows which.
Thanks,
-Steve
The status is simply for your reference and doesn't represent the actual presence of data in the repository. You could achieve a similar thing simply by clearing all data in the repository without recreating it.
If you really need to rely on those status records you can clear the status for a given file once you polled it and determined it's done (or prior to starting an import) with this curl:
curl -X DELETE http://127.0.0.1:7200/rest/data/import/server/test1/status \
-H 'content-type: application/json' -d '["SysML.owl"]'
Note that this is an undocumented API and it may change without notice.

How can this curl command be converted to pycurl or python requests?

any hel pwill be highly appreciated. Just can't make this to work. I am basically trying to update my Grafana dashboard via a http request. managed to get it to work with curl but want to do this with python's requests or pycurl.
curl -X PUT https://<api token>#ks.hostedgraphite.com/api/v2/grafana/dashboards/<my_dashboard> --data-binary #dashboard.json
The command above works. Tried several ways, an example of a code snippet:
crl = pycurl.Curl()
crl.setopt(crl.URL,
'https://apitoken#ks.hostedgraphite.com/api/v2/grafana/dashboards/<my_dashborad>')
crl.setopt(crl.UPLOAD, 1)
file = open('dashboard.json')
crl.setopt(crl.READDATA, file)
crl.perform()
crl.close()
file.close()
print('Status: {}'.format(crl.getinfo(crl.RESPONSE_CODE)))
curl -X PUT https://api_token#ks.hostedgraphite.com/api/v2/grafana/dashboards/my_dashboard --data-binary #dashboard.json
translates to
import requests
data = open('dashboard.json', 'rb').read()
response = requests.put('https://api_token#ks.hostedgraphite.com/api/v2/grafana/dashboards/my_dashboard', data=data)
Just replace proper values for api_token and my_dashboard.
You can use https://curl.trillworks.com/ for converting curl commands to equivalent code using python requests.
ok, so my main issue was creating a http request via python requests.
Figured it out, I setthe auth param to auth=("graphite_api_token","") and it worked:
data = json.dumps(dashbord_dict)
headers = {"Accept": "application/json","Content-Type":"application/json"}
response =requests.put('https://ks.hostedgraphite.com/api/v2/grafana/dashboards/some_dashboard,auth=("graphite_api_token",""),data=data,headers=headers)
print(response.status_code)

How to retrieve Jira issues via REST API when query contains a space

I'm trying to retrieve all issues from a project that are status "New Ticket". The status name having a space in it is completely tripping me up.
My command looks like the following. This works perfectly if status is a single word, like Closed, but fails on the spacing in New Ticket.
curl -D- \
-u user:password \
-X POST \
-H "Content-Type: application/json" \
--data '{"jql":"project = DEVOPS AND status = New Ticket", "fields":["id","key","summary","description"]}' \
"https://projecturl.atlassian.net/rest/api/2/search"
I've tried the following with no success;
"project = DEVOPS AND status = New Ticket"
"project = DEVOPS AND status = 'New Ticket'"
"project = DEVOPS AND status = %22New Ticket%22"
"project = DEVOPS AND status = \\u0025New Ticket\\u0025"
What's the right syntax?
After a bunch of messing around, it turns out you can just escape quotes inside the query, like so;
"project = DEVOPS AND status = \"New Ticket\""
What about if it's a GET request and the request is in the URI, not in the POST data?
I have something similar to the following
/usr/bin/curl \
-s -u user#company.com:APIKEY -X GET \
-H "Content-Type: application/json" \
https://company.atlassian.net/rest/api/2/search?expand=changelog&jql=project=KEY%20and%20issueType=BUG
For me it's lopping off the %20and%20issueType=BUG part (and presumably dropping an error to STDERR and running it as if the jql is just project=KEY
EDIT:
This is starting to feel like a bug, even though it works just fine when you paste the URI into the browser.
Replacing the %20 with + made no difference. I get the same results either way.
The issue seems to be the presence of the expand=changelog and a JQL with spaces in it together. When I remove it, it works fine. When I put it at the end so that I have https://company.atlassian.net/rest/api/2/search?jql=project=KEY+and+issueType=BUG&expand=changelog instead the JQL part works, and I get the a query of all the BUGs in KEY, but it ignores the expand=changelog part and I don't get the bug history of the bugs.
I was using Ansible to open a ticket via jira!
To solve the error related to space between the words I just replace the space to "+" to replace the spaces, follow i.g:
Maybe this can help you ;)
- name: Checking if the Jira issue already exist
uri:
url: "{{ jira_server }}rest/api/3/search?jql=text~'{{ jira_subject | replace(' ','+')}}{{ jira_fw_type_operation }}'&project={{ jira_project }}"
user: "{{ jira_user }}"
password: "{{ jira_token }}"
force_basic_auth: yes
return_content: yes
status_code: 200
timeout: 60
headers:
Accept: "application/json"
register: fetch_issue
changed_when: fetch_issue['json']['total'] == 0

Dockerized DCTM 7.3 and Dockerized DCTM REST 7.3 not able to retrieve global registry or its documents

My setup consists of
Documentum Content Server 7.3 (dctm-cs) running in a docker container (from EMC)
Documentum REST Services 7.3 (dctm-rest) running in a docker container (from EMC)
I am definitively able to get information from within dctm by running queries against it with iapi, for example:
API> ?,c,select user_name from dm_user enable (return_top 5)
user_name
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
docu
ubuntudb
dm_superusers
dm_superusers_dynamic
dm_browse_all
(5 rows affected)
I am also able to $ curl http://localhost:8080/dctm-rest/repositories.json from both the dctm-rest container as well as its host container and get the results:
{"id":"http://localhost:8080/dctm-rest/repositories","title":"Repositories","author":[{"name":"EMC Documentum"}],"updated":"2017-08-16T21:42:44.177+00:00","page":1,"items-per-page":1000,"total":1,"links":[{"rel":"self","href":"http://localhost:8080/dctm-rest/repositories.json"}],"entries":[{"id":"http://localhost:8080/dctm-rest/repositories/ubuntudb","title":"ubuntudb","summary":"ubuntudb","updated":"2017-08-16T21:42:44.178+00:00","published":"2017-08-16T21:42:44.178+00:00","links":[{"rel":"edit","href":"http://localhost:8080/dctm-rest/repositories/ubuntudb.json"}],"content":{"type":"application/json","src":"http://localhost:8080/dctm-rest/repositories/ubuntudb.json"}}]}
Attempting to $ curl http://localhost:8080/dctm-rest/repositories/ubuntudb.json, however hangs indefinitely.
I have attempted to provide the default username and password via basic HTTP authentication, also with the same results.
The contents of the dfc.properties file in dctm-cs:
dfc.data.dir=/opt/dctm
dfc.tokenstorage.dir=/opt/dctm/apptoken
dfc.tokenstorage.enable=false
dfc.docbroker.host[0]=ubuntustateless
dfc.docbroker.port[0]=1489
dfc.crypto.repository=ubuntudb
dfc.session.secure_connect_default=try_secure_first
dfc.globalregistry.repository=ubuntudb
dfc.globalregistry.username=dm_bof_registry
dfc.globalregistry.password=AAAAEL9wp8c6k3K2UTQJwTYO5kMnE3rDrHJVDL+LijAg+zLk
The contents of the dfc.properties file in dctm-rest:
dfc.docbroker.host[0]=172.18.0.1
dfc.docbroker.port[0]=1489
#Add the global registry repository name to the following key.
dfc.globalregistry.repository=ubuntudb
#Add the username of the global registry user to the following key.
dfc.globalregistry.username=dmadmin
#Add an encrypted password value for the following key.
dfc.globalregistry.password=password
dfc.exception.include_id=false
dfc.exception.include_decoration=false
I have attempted to change the value of dfc.globalregistry.username to be the same as in dctm-cs, to no avail and same hang on request.
I have also attempted to use both encrypted and decrypted values for dfc.globalregistry.password, in both dctm-cs and dctm-rest also with no luck.

Is there a way to add additional configurable settings in OpsCenter 6.0.2 Lifecycle Manager config profiles?

I would really like to add the following settings to our spark-defaults.conf using OpsCenter 6.0.2 in order to avoid configuration drift. Is there a way to add these config items to the config profile template?
spark.cores.max 4
spark.driver.memory 2g
spark.executor.memory 4g
spark.python.worker.memory 2g
NOTE: As Mike Lococo has pointed out in the comments for this answer -- this answer may work to update the config profile values but will not result in those values being written to spark-defaults.conf.
The following is not a solution!
You can; you have to update the config profile via the LCM Config Profile API (https://docs.datastax.com/en/opscenter/6.0/api/docs/lcm_config_profile.html#lcm-config-profile).
First, identify the config profile that needs updating:
$ curl http://localhost:8888/api/v1/lcm/config_profiles
Get the href for the specific config profile that needs updating, request it, and save the response body to a file:
$ curl http://localhost:8888/api/v1/lcm/config_profiles/026fe8e3-0bb8-49c1-9888-8187b1624375 > profile.json
Now, in the profile.json file you just saved to, you add or edit the key at json > spark-defaults-conf to include the following keys:
"spark-defaults-conf": {
"spark-cores-max": 4,
"spark-python-worker-memory": "2g",
"spark-ssl-enabled": false,
"spark-drivers-memory": "2g",
"spark-executor-memory": "4g"
}
Save the updated profile.json. Finally, execute an HTTP PUT to the same config profile URL, using the edited file as the request data:
$ curl -X PUT http://localhost:8888/api/v1/lcm/config_profiles/026fe8e3-0bb8-49c1-9888-8187b1624375 -d #profile.json