I have approximately 250,000 JSON-formatted files, each with one object in it (formatted just how CouchDB likes it with _id). What's the best way to import these into my remote CouchDB server as records?
-I am on a windows xp machine.
-I have internet access but I can't set up a couchDB server on my local machine and have it be WWW accessible (firewall constraints.) so no easy replication.
I would highly suggest that you look into the bulk doc API in the couchdb wiki: http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API
Basically, you make a POST request to /someDatabase/_bulk_docs that looks like this:
{
"docs": [
{ "_id": "awsdflasdfsadf", "foo": "bar" },
{ "_id": "cczsasdfwuhfas", "bwah": "there" },
...
]
}
Just like any other POST request, if you don't include _id properties, couchdb will generate them for you.
You can use this same operation to update a bunch of docs: just include their _rev property. And if you want to delete any of the docs that you are updating, then add a "_deleted": true property to the document.
If you have a json file with your documents and use curl, it could look like:
curl -H "Content-Type: application/json" --data-binary #/home/xxx/data.json https://usr:pwd#host:5984/someDatabase/_bulk_docs/
Cheers.
Related
I am trying to enable kv secret engine at secret path in my vault setup..
I can easily do it using CLI
vault secrets enable -path=secret kv
But I have to make it work using Vault's HTTP APIs.
I have gone through documentation but could not find any endpoint for the above command.
Thanks in advance
This is covered under the System Backend/sys/mounts API reference page.
Issue a POST request to /v1/sys/mounts/<mountpoint> with a payload containing the type (kv) and various configuration options. For KV, you probably want to specify version: 2 (or type kv-v2) unless you want to stick to V1.
See the above link for details on the possible parameters.
Here is the example from the docs:
payload.json:
{
"type": "aws",
"config": {
"force_no_cache": true
}
}
Request:
$ curl \
--header "X-Vault-Token: ..." \
--request POST \
--data #payload.json \
http://127.0.0.1:8200/v1/sys/mounts/my-mount
Been going through the documentation (https://cdn.rawgit.com/rabbitmq/rabbitmq-management/v3.7.9/priv/www/api/index.html)
And did not find a way to publish a message to a queue (not an exchange, a queue) with the HTTP API?
Is that possible?
As much as it might make little sens in a production mindset, it still can be useful for testing purposes.
I basically want to mimic the “Publish message” interface available in the RabbitMQ administration console.
Is this possible somehow?
Note: your question is already answered here: link
RabbitMQ only supports publishing to exchanges. This is a core function of the product. Then, RabbitMQ uses bindings to figure out what queue(s) should receive the message.
You can read about these concepts here.
When you use "Publish message" in the administration console, it uses the default binding and default exchange. From this document:
The default exchange
In previous parts of the tutorial we knew nothing about exchanges, but still were able to send messages to queues. That was possible because we were using a default exchange, which we identify by the empty string ("").
Recall how we published a message before:
channel.basic_publish(exchange='',
routing_key='hello',
body=message)
The exchange parameter is the name of the exchange. The empty string denotes the default or nameless exchange: messages are routed to the queue with the name specified by routing_key, if it exists.
So, in order to appear to publish directly to a queue, the management interface publishes a message to the default exchange (named amq.default or the empty string "") using the queue name as the routing key. You can see this for yourself by enabling the developer tools in your browser and watching the HTTP call made to /api/exchanges/vhost/name/publish when you publish a message to a queue.
In your case, the request will look something like this (use Chrome, and right-click the publish request and "copy as cUrl"):
curl -4vvv -u guest:guest \
'localhost:15672/api/exchanges/%2F/amq.default/publish' \
-H 'Content-Type: text/plain;charset=UTF-8' \
--data-binary '{"vhost":"/","name":"amq.default","properties":{"delivery_mode":1,"headers":{}},"routing_key":"MY-QUEUE-NAME","delivery_mode":"1","payload":"TEST","headers":{},"props":{},"payload_encoding":"string"}'
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
For those interested in Intellij IDEA HTTP Client syntax with an array of ids
[
{"id": "83d6e4dc-0478-42da-8da0-65b508530a43"},
{"id": "08d3e147-79c4-4b91-be7c-b1cc86e21278"}
]
POST http://localhost:15672/api/exchanges/%2F/amqp.myexchange/publish
Authorization: Basic guest guest
Content-Type: application/json
{
"vhost": "/",
"name": "amqp.myexchange",
"properties": {
"delivery_mode": 2,
"headers": {},
"content_type": "application/json"
},
"routing_key": "",
"delivery_mode": "2",
"payload": "[{\"id\":\"83d6e4dc-0478-42da-8da0-65b508530a43\"},{\"id\":\"08d3e147-79c4-4b91-be7c-b1cc86e21278\"}]",
"headers": {},
"props": {
"content_type": "application/json"
},
"payload_encoding": "string"
}
I know this post is quite old, but maybe still get found.
Here is a 3rd party tool that may help in such situations, when you need to publish a message directly into a queue:
https://github.com/bkrieger1991/rabbitcli
C:\Program Files\RabbitMQ Server\rabbitmq_server-3.7.18\sbin>
C:\Program Files\RabbitMQ Server\rabbitmq_server-3.7.18\sbin>curl -i -u guest:guest localhost:15672/api/exchanges/%2F/amq.default/publish -H 'content-type:application/json" -d '{"vhost":"/","name":"amq.default","properties":{"delivery_mode":1,"headers":{}},"routing_key":"TEST,"delivery_mode":"1","payload":"TEST","headers":{},"props":{},"payload_encoding":"string"}'
HTTP/1.1 405 Method Not Allowed
allow: POST, OPTIONS
content-length: 0
content-security-policy: default-src 'self'
date: Fri, 06 Dec 2019 14:03:08 GMT
server: Cowboy
vary: origin
I'm testing Elasticsearch in development mode with docker official image.
The basic install is based on X_pack and basic authentication.
Everything works fine by performing curl like:
curl -XPUT -u elastic:elasticpassword "http://localhost:9200/movies/movie/1" -d'
{
"title": "The Godfather",
"director": "Francis Ford Coppola",
"year": 1972, "user":"elastic", "password":"changeme"
}'
But is there a way to perform a token request (with user and password) and then query Elasticsearch with the token. Instead of having to specify user/password every time I perform a query?
The default X_Pack in docker image has Basic authentication enabled. Which is what your are using. The token for the same is base64(user:password). You can generate the same using http://base64encode.org and inputing :.
In curl there are two ways to call Basic auth URLs
curl -XPUT -u elastic:elasticpassword "http://localhost:9200/movies/movie/1" -d''
which you have already been using
curl -H "Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==" -XPUT "http://localhost:9200/movies/movie/1" -d'....'
Now if your problem is putting in this again and again then you better create a alias in your bash profile like below
alias curles='curl -u elastic:elasticpassword'
After that you can call your commands as below
curles -XPUT "http://localhost:9200/movies/movie/1" -d''
Cutting out a lot of my original answer because you could argue it's all local, but leaving one major complaint about security here:
Definitely don't go to base64encode.org and enter your username:password. The very fact that it is suggested here on StackOverflow makes that site now (sadly for the owners) and incredibly juicy hacking target to compromise since they know people are going there.
curl -X GET --user myuser:mypassword "http://elasticsearch:9200/_cluster/health?pretty"
In my case above curl helped
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 16,
"active_shards" : 16,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Further to Taran's answer, if you want to generate base64 token, you can do so by the following command:
echo -n 'username:pass' | openssl base64
I am using the Embeddable Reporting service for Bluemix. I want to dynamically change the SQL text in the SQLQuery of a report definition so that the report is generated from a referenced table. The report specification can be changed using the https://erservice-impl.ng.bluemix.net:443/ers/v1/definitions//specificationAPI. However, how do you change the SQL text?
Here are the steps that are involved:
Author a report in the your favorite authoring tool against a sample table.
Grab the report specification by clicking Tools > Copy Report to Clipboard.
Locate the SQL string in the report specificiation and change it to a token that you can easily search/replace. For example: Select * from %%DATATABLE%%
Save this template in a file or in a string variable within your app.
Search/replace the string %%DATATABLE%% with the table that you want.
Execute the report:
a. POST to /ers/v1/packages/(package-id)/reports with content-type application/xml and the specification as the body. The response will contain the location of the new report.
b. GET /ers/v1/reports/(report-id)/html to render the report in HTML.
Here is an example in curl:
Connect to the service with your cloudant/Mongo URL and Embeddable Reporting Service credentials. For example:
curl --insecure -i -X POST --cookie-jar newcookies.txt -H "Content-Type: application/json" -d '{"bundleUri":"https:/(repository-uri)/"}' https://(ers-userid):(ers-password)#erservice-impl.ng.bluemix.net/ers/v1/connection
Create a run instance from a specification against the packageId that contains your database.For example:
curl --insecure -i -X POST -b newcookies.txt -H "Content-Type: application/xml" -d '(report-spec-goes-here)' https://erservice-impl.ng.bluemix.net/ers/v1/packages/(package-id)/reports
Here is an example response:
{
...
"_links": {
"formats": [
{
"href": "/ers/v1/reports/JwVd3JkeR3SFUkLlOi1bWw/html",
"disposition": "paged",
"mimeType": "text/html"
},
... other formats ...
]
},
"_bundleID": "JwVd3JkeR3SFUkLlOi1bWw"
}
You can now run the response._links.formats.href
Run the instance in HTML. For example:
curl --insecure -i -X GET -b newcookies.txt https://erservice-impl.ng.bluemix.net:443/ers/v1/reports/JwVd3JkeR3SFUkLlOi1bWw/html
Get the HTML back.
I'm using AWS OpsWorks for a Rails application with Redis and Sidekiq and would like to do the following:
Override the maxmemory config for redis
Only run Redis & Sidekiq on a selected EC2 instance
My current JSON config only has the database.yml overrides:
{
"deploy": {
"appname": {
"database": {
"username": "user",
"password": "password",
"database": "db_production",
"host": "db.host.com",
"adapter": "mysql2"
}
}
}
}
Override the maxmemory config for redis
Take a look and see if your Redis cookbook of choice gives you an attribute to set that / provide custom config values. I know the main redisio one lets you set config value, as I do it on my stacks (I set the path to the on disk cache, I believe)
Only run Redis & Sidekiq on a selected EC2 instance
This part is easy: create a Layer for Redis (or Redis/Sidekiq) and add an instance to that layer.
Now, because Redis is on a different instance than your Rails server, you won't necessarily know what the IP address for your Redis server is. Especially since you'll probably want to use the internal EC2 IP address vs the public IP address for the box (using the internal address means you're already inside the default firewall).
Sooo... what you'll probably need to do is to write a custom cookbook for your app, if you haven't already. In your attributes/default.rb write some code like this:
redis_instance_details = nil
redis_stack_name = "REDIS"
redis_instance_name, redis_instance_details = node["opsworks"]["layers"][redis_stack_name]["instances"].first
redis_server_dns = "127.0.0.1"
if redis_instance_details
redis_server_dns = redis_instance_details["private_dns_name"]
end
Then later in the attributes file set your redis config to your redis_hostname (maybe using it to set:
default[:deploy][appname][:environment_variables][:REDIS_URL] = "redis://#{redis_server_dns}:#{redis_port_number}"
Hope this helps!