curl -i -X PUT -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/sink-s3-voluble/config \
-d '
{
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"key.converter":"io.confluent.connect.protobuf.ProtobufConverter",
"tasks.max": "1",
"topics": "cats",
"s3.region": "ap-south-1",
"s3.bucket.name": "testbucketkafka",
"flush.size": "65536",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator",
"schema.compatibility": "NONE",
"partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
"transforms": "AddMetadata",
"transforms.AddMetadata.type": "org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddMetadata.offset.field": "_offset",
"transforms.AddMetadata.partition.field": "_partition"
}
The error given is Connector configuration is invalid and contains the following 1 error(s):\nInvalid value io.confluent.connect.s3.format.protobuf.ProtobufFormat for configuration format.class: Class io.confluent.connect.s3.format.protobuf.ProtobufFormat could not be found.\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}
The error is correct. There is no config.format=ProtobufFormat configuration
https://docs.confluent.io/kafka-connectors/s3-sink/current/configuration_options.html#connector
Set format.class to a supported value.
This call:
curl -X POST \
-H "Authorization: key=AAAAx4uYEXU:...e8EEq9aCv-pGWXIfyCp_lUp28mY" \
-H "Content-Type: application/json" \
-d '{ "data": {
"score": "5x1",
"time": "15:10"
},
"condition": "'dogs' in topics || 'cats' in topics",
"content_available": true
}' \
"https://fcm.googleapis.com/fcm/send"
Causes this error:
Invalid "condition" field: only support 'topics' conditions
I copied the condition field straight from the documentation... how can I get this to work?
I want to create and start a docker container with some arguments (to the CMD) through docker API using a json file.
curl -XPOST --unix-socket /var/run/docker.sock -d #minio.json -H 'Content-Type: application/json' http://localhost/containers/create?name=minio
curl -XPOST --unix-socket /var/run/docker.sock -H 'Content-Type: application/json' http://localhost/containers/minio/start
I tried to put them in the top of the json and the HostConfig "sub entry"
"Args": "server --address 192.168.150.3:80 /export",
but when I inspect the arguments are not there.
In the documentation there are examples only when you inspect a container and see the arguments.
Is there a way to pass Args within the json file or when you start the container with curl and a payload ("docker run" is excluded) ?
UPDATE:
Solution is to list all arguments in double quotes in the array.
"Cmd": [
"server" ,"--address", "192.168.100.8:8888", "/export"
],
The API has two attributes in JSON
"Cmd": [
"date"
],
"Entrypoint": "",
You need to change the Cmd array to pass the arguments you want
i have problem with the API of ICINGA2.
i'm trying to add new variables with the POST call ,
i'm getting the required result,
But ICINGA2 didn't add the new var.
According to documentation:
http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/icinga2-api
With the following API, i creates all our hosts in vienna :
curl -k -s -u root:icinga -H 'Accept: application/json' -X PUT 'https://localhost:5665/v1/objects/hosts/server.example.com' \
-d '{ "templates": [ "generic-host" ], "attrs": { "zone": "Vienna", "address": "180.33.1.123", "check_command": "hostalive", "vars.os" : "Linux", "vars.agent" : "ssh" } }' \
| python -m json.tool
While this part works as expected,
The problem is once host created, i need to add various vars for different servers.
for example
Adding of the variable: "vars.servicename" : "DHCP_Servers"
If i'm going back to the documentation, the below API that will need to be execute:
curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/objects/hosts/server.example.com' \
-d '{ "templates": [ "generic-host" ], "attrs": { "zone": "Vienna", "address": "180.33.1.123", "check_command": "hostalive", "vars.os" : "Linux", "vars.agent" : "ssh", "vars.servicename" : "DHCP_Servers" } }' \
| python -m json.tool
When i ran the API , as expected i'm getting back:
{
"results": [
{
"code": 200.0,
"name": "server.example.com",
"status": "Attributes updated.",
"type": "Host"
}
]
}
But there is no changes that taking place on ICINGA/ host file.
Obviously the same user as in my inbox and the forums (https://monitoring-portal.org/index.php?thread/37160-adding-vars-with-api/&postID=234885#post234885) lately. Leaving this as a note here as it might help others to see why it does not work. That feature is just not implemented as it involves storing the applied changes, do a rollback, and re-apply. Not as simple as it sounds.
https://dev.icinga.org/issues/11501
I have typed this command to index a document in Elasticsearch
create an index
curl -X PUT "localhost:9200/test_idx_1x"
create a mapping
curl -X PUT "localhost:9200/test_idx_1x/test_mapping_1x/_mapping" -d '{
"test_mapping_1x": {
"properties": {
"my_attachments": {
"type": "attachment"
}
}
}
}'
index this document
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/4' -d '{
"post_date": "2009-11-15T14:12:12",
"message": "test Elastic Search",
"name": "N1"
}'
All these three commands are very goods.
But when I type this command:
curl -XPOST 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
"post_date": "2009-11-15T14:12:12",
"message": "trying out Elastic Search",
"name": "N2",
"my_attachments": {
"type": "attachment",
"_content_type": "text/plain",
"file": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt"
}
}'
I receive this error message:
{
"error": "NullPointerException[null]",
"status": 500
}
I change it into;
curl -XPOST 'http://localhost:9200/test_idx_1x/test_mapping_1x/1bis' -d '{
"post_date": "2009-11-15T14:12:12",
"message": "trying out Elastic Search",
"name": "N2",
"my_attachments": {
"type": "attachment",
"_content_type": "text/plain",
"_name": "/inf/bd/my_home_directory/test.txt"
}
}'
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
"post_date": "2009-11-15T14:12:12",
"message": "trying out Elastic Search",
"name": "N2",
"my_attachments": {
"file": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt"
}
}'
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
"post_date": "2009-11-15T14:12:12",
"message": "trying out Elastic Search",
"name": "N2",
"my_attachments": {
"file": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt",
"_content_type": "text/plain"
}
}'
The output is the same error.
I change it like that
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
"user": "kimchy",
"post_date": "2009-11-15T14:12:12",
"message": "trying out Elastic Search",
"name": "N2",
"my_attachments": {
"file": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt",
"_content_type": "text/plain",
"content": "... base64 encoded attachment ..."
}
}'
the error is
{
"error": "MapperParsingException[Failed to parse]; nested: JsonParseException[Failed to decode VALUE_STRING as base64 (MIME-NO-LINEFEEDS): Illegal character '.' (code 0x2e) in base64 content\n at [Source: [B#159b3; line: 1, column: 241]]; ",
"status": 400
}
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
"post_date": "2009-11-15T14:12:12",
"message": "trying out Elastic Search",
"name": "N2",
"my_attachments": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt"
}'
I receive this error message:
{
"error": "MapperParsingException[Failed to parse]; nested: JsonParseException[Unexpected character ('h' (code 104)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: [B#1ae9565; line: 1, column: 132]]; ",
"status": 400
}
if I type
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
"post_date": "2009-11-15T14:12:12",
"message": "trying out Elastic Search",
"name": "N2",
"my_attachments": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt"
}'
I receive error. I can understand it
{
"error": "MapperParsingException[Failed to parse]; nested: JsonParseException[Failed to decode VALUE_STRING as base64 (MIME-NO-LINEFEEDS): Illegal character ':' (code 0x3a) in base64 content\n at [Source: [B#1ffb7d4; line: 1, column: 137]]; ",
"status": 400
}
How can I use attach files to ES so that ES can index it?
Thanks for your answer. That attachment plugin I have already installed when I type these commands. The content of the text file is encoded in Base64, so I don't encode it anymore. If I don't use the file's path but directly use its contents in Base 64, ex.
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/' -d '{
"post_date": "2009-11-15T14:12:12",
"message": "trying out Elastic Search",
"name": "N2",
"my_attachments": "file's content string encoded in base64"
}'
all is good, I have already succeeded in posting file and searching its content later.
But if I replace it with path's file, I obtained negative results. So I want to know how to encode Base64 a file in command line,in the command of ES indexing (of course, I don't want to type base64 command to encode a file before typing 2nd command to indexing it in ES). As your answer, do I have to installed something like "Perl library" to execute your command?
http://es-cn.medcl.net/tutorials/2011/07/18/attachment-type-in-action.html
#!/bin/sh
coded=`cat fn6742.pdf | perl -MMIME::Base64 -ne 'print encode_base64($_)'`
json="{\"file\":\"${coded}\"}"
echo "$json" > json.file
curl -X POST "localhost:9200/test/attachment/" -d #json.file
First, you don't specify whether you have the attachment plugin installed. If not, you can do so with:
./bin/plugin -install mapper-attachments
You will need to restart ElasticSearch for it to load the plugin.
Then, as you do above, you map a field to have type attachment:
curl -XPUT 'http://127.0.0.1:9200/foo/?pretty=1' -d '
{
"mappings" : {
"doc" : {
"properties" : {
"file" : {
"type" : "attachment"
}
}
}
}
}
'
When you try to index a document, you need to encode the contents of your file in Base64. You could do this on the command line using the base64 command line utility. However, to be legal JSON, you also need to encode new lines, which you can do by piping the output from base64 through Perl:
curl -XPOST 'http://127.0.0.1:9200/foo/doc?pretty=1' -d '
{
"file" : '`base64 /path/to/file | perl -pe 's/\n/\\n/g'`'
}
'
Now you can search your file:
curl -XGET 'http://127.0.0.1:9200/foo/doc/_search?pretty=1' -d '
{
"query" : {
"text" : {
"file" : "text to look for"
}
}
}
'
See ElasticSearch attachment type for more.
This is a complete shell script implementation:
file_path='/path/to/file'
file=$(base64 $file_path | perl -pe 's/\n/\\n/g')
curl -XPUT "http://eshost.com:9200/index/type/" -d '{
"file" : "content" : "'$file'"
}'
There is an alternative solution - plugin at http://elasticwarehouse.org. You can upload binary file using _ewupload?, read newly generated ID and update your different index with this reference.
Install plugin:
plugin -install elasticwarehouseplugin -u http://elasticwarehouse.org/elasticwarehouse/elasticsearch-elasticwarehouseplugin-1.2.2-1.7.0-with-dependencies.zip
Restart cluster, then:
curl -XPOST "http://127.0.0.1:9200/_ewupload?folder=/myfolder&filename=mybinaryfile.bin" --data-binary #mybinaryfile.bin
Sample response:
{"id":"nWvrczBcSEywHRBBBwfy2g","version":1,"created":true}