I am trying to restart Mapreduce Jobtracker through Cloudera Manager API. Stats for Jobtracker is as follows :
local-iMac-399:$ curl -u 'admin:admin' 'http://hadoop-namenode.dev.com:7180/api/v6/clusters/Cluster%201/services/mapreduce/roles/mapreduce-JOBTRACKER-0675ebab2b87e3869e0d90167cf4bf86'
{
"name" : "mapreduce-JOBTRACKER-0675ebab2b87e3869e0d90167cf4bf86",
"type" : "JOBTRACKER",
"serviceRef" : {
"clusterName" : "cluster",
"serviceName" : "mapreduce"
},
"hostRef" : {
"hostId" : "24259373-7e71-4089-8251-faf055e42ad7"
},
"roleUrl" : "http://hadoop-namenode.dev.com:7180/cmf/roleRedirect/mapreduce-JOBTRACKER-0675ebab2b87e3869e0d90167cf4bf86",
"roleState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "JOB_TRACKER_FILE_DESCRIPTOR",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_GC_DURATION",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_HOST_HEALTH",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_LOG_DIRECTORY_FREE_SPACE",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_SCM_HEALTH",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_UNEXPECTED_EXITS",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_WEB_METRIC_COLLECTION",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "STALE",
"haStatus" : "ACTIVE",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"commissionState" : "COMMISSIONED",
"roleConfigGroupRef" : {
"roleConfigGroupName" : "mapreduce-JOBTRACKER-BASE"
}
}
local-iMac-399:$
Dont know How do I use API to restart just Jobtracker ?
I tried to restart Hive service using following command but got some error
local-iMac-399:$curl -X POST -u 'admin:admin' 'http://hadoop-namenode.dev.com:7180/api/v6/clusters/Cluster%201/services/hive/roleCommands/restart'
{
"message" : "No content to map due to end-of-input\n at [Source: org.apache.cxf.transport.http.AbstractHTTPDestination$1#4169c499; line: 1, column: 1]"
}
I would appreciate if someone help in understanding how to use Cloudera Manager API
Based on the information provided, this is how you'd invoke the CM API JobTracker restart
curl -u 'admin:admin' -X POST -H "Content-Type:application/json" -d '{"items":["mapreduce-JOBTRACKER-0675ebab2b87e3869e0d90167cf4bf86"]}' 'http://hadoop-namenode.dev.com:7180/api/v6/clusters/Cluster%201/services/mapreduce/roleCommands/restart'
Related
I need to GET list of nodes that contains certain label.
I know how to do that by getting entire nodes list by using Jenkins REST API and then getting node by node also REST API and checking its labels - but its to many API calls.
I also can create some job that writing to some place nodes list by label as parameter - but its bad way as Jenkins job that triggered remotely have no return value and I cant know it finished and will need read results from some other place the job saved it there.
I need some way that by one API call I will get nodes list contains a given label.
You can run a single API call to <JENKINS_URL>/computer/api/json (or <JENKINS_URL>/computer/api/python for a python api) which return a list of all nodes and their properties.
One of the properties is the label - so just go over all nodes and extract the ones that contain your needed label.
Here is an example for the returned object:
{
"_class" : "hudson.model.ComputerSet",
"busyExecutors" : 0,
"computer" : [
{
"_class" : "hudson.model.Hudson$MasterComputer",
"actions" : [
],
"assignedLabels" : [
{
"name" : "built-in"
}
],
"description" : "the Jenkins controller's built-in node",
"displayName" : "Built-In Node",
"executors" : [
{
},
{
}
],
"icon" : "symbol-computer",
"iconClassName" : "symbol-computer",
"idle" : true,
"jnlpAgent" : false,
"launchSupported" : true,
"loadStatistics" : {
"_class" : "hudson.model.Label$1"
},
"manualLaunchAllowed" : true,
"monitorData" : {
"hudson.node_monitors.SwapSpaceMonitor" : {
"_class" : "hudson.node_monitors.SwapSpaceMonitor$MemoryUsage2",
"availablePhysicalMemory" : 6938730496,
"availableSwapSpace" : 6906019840,
"totalPhysicalMemory" : 16885276672,
"totalSwapSpace" : 21046026240
},
"hudson.node_monitors.TemporarySpaceMonitor" : {
"_class" : "hudson.node_monitors.DiskSpaceMonitorDescriptor$DiskSpace",
"timestamp" : 1653907906021,
"path" : "C:\\Windows\\Temp",
"size" : 426696622080
},
"hudson.node_monitors.DiskSpaceMonitor" : {
"_class" : "hudson.node_monitors.DiskSpaceMonitorDescriptor$DiskSpace",
"timestamp" : 1653907905929,
"path" : "C:\\ProgramData\\Jenkins\\.jenkins",
"size" : 426696622080
},
"hudson.node_monitors.ArchitectureMonitor" : "Windows 10 (amd64)",
"hudson.node_monitors.ResponseTimeMonitor" : {
"_class" : "hudson.node_monitors.ResponseTimeMonitor$Data",
"timestamp" : 1653907905941,
"average" : 0
},
"hudson.node_monitors.ClockMonitor" : {
"_class" : "hudson.util.ClockDifference",
"diff" : 0
}
},
"numExecutors" : 2,
"offline" : false,
"offlineCause" : null,
"offlineCauseReason" : "",
"oneOffExecutors" : [
],
"temporarilyOffline" : false
}
],
"displayName" : "Nodes",
"totalExecutors" : 2
}
You are interested in the assignedLabels object - notice that it can contain multiple labels.
I was following this tutorial to setup a configmap for a redis.conf. After I create the Redis deployment, I check to ensure that the redis.conf file is in each of the pods, and they are there. The problem is that when go in the redis-cli and check the configuration there, the redis.conf values aren't used. The default values are being used as if the Redis did not start up with the redis.conf file.
redis.conf
maxclients 2000
requirepass "test"
redis-config configmap
{
"apiVersion": "v1",
"data": {
"redis-config": "maxclients 2000\nrequirepass \"test\"\n\n"
},
"kind": "ConfigMap",
"metadata": {
"creationTimestamp": "2018-03-07T15:28:19Z",
"name": "redis-config",
"namespace": "default",
"resourceVersion": "2569562",
"selfLink": "/api/v1/namespaces/default/configmaps/redis-config",
"uid": "29d250ea-221c-11e8-969f-06c0c8d545d2"
}
}
k8 redis manifest.json
{
"kind" : "Deployment",
"apiVersion" : "extensions/v1beta1",
"metadata" : {
"name" : "redis-master",
"creationTimestamp" : null
},
"spec" : {
"replicas" : 2,
"template" : {
"metadata" : {
"creationTimestamp" : null,
"labels" : {
"app" : "redis",
"role" : "master",
"tier" : "backend"
}
},
"spec" : {
"hostNetwork" : true,
"nodeSelector" :{ "role": "cache"},
"containers" : [{
"name" : "master",
"image" : "redis",
"ports" : [{
"containerPort" : 6379,
"hostPort" : 6379,
"protocol" : "TCP"
}
],
"volumeMounts" : [{
"mountPath" : "/redis-master",
"name": "config"
}
],
"resources" : {},
"terminationMessagePath" : "/dev/termination-log",
"imagePullPolicy" : "IfNotPresent"
}],
"volumes" : [{
"name" : "config",
"configMap" : {
"name" : "redis-config",
"items": [{
"key": "redis-config",
"path": "redis.conf"
}]
}
}
],
"restartPolicy" : "Always",
"terminationGracePeriodSeconds" : 30,
"dnsPolicy" : "ClusterFirst",
"securityContext" : {}
}
}
},
"status" : {}
}
Now I know the tutorial uses a Pod kind, and I am using a Deployment kind, but I don't think that is the issue here.
It looks like you are pulling the default redis container. If you check the redis Dokerfiles, for example https://github.com/docker-library/redis/blob/d53b982b387634092c6f11069401679034054ecb/4.0/alpine/Dockerfile, at the bottom, they have:
CMD ["redis-server"]
which will start redis with the default configuration.
Per redis documentation:
https://redis.io/topics/quickstart
under "Starting Redis" section, if you want to provide a different configuration, you would need to start redis with:
redis-server <config file>
Additionally the example in Kubernetes documentation uses a different redis containter:
image: kubernetes/redis
And from the Dokerfile: https://github.com/kubernetes/kubernetes/blob/master/examples/storage/redis/image/Dockerfile, it seems like that one starts Redis with the provided configuration.
I want to be able to extract content from a PDF file and to be able to search within that content using ElasticSearch.
I did install elasticsearch/elasticsearch-mapper-attachments/2.6.0
I have created a new index named "docs".
I did create a file named "tmp.json" with that content :
{"title": "file.pdf", "file": "IkdvZCBTYXZlIHRoZSBRdWVlbiIgKGFsdGVybmF0aXZlbHkgIkdvZCBTYXZlIHRoZSBLaW5nIg=="}
I did execute the following :
curl -X PUT "http://localhost:9200/docs/attachment/_mapping" -d '{
"attachment": {
"properties" : {
'file" : {
"type" : "attachment",
"fields" : {
"title" : {"store":"yes"},
"file":{
"type":"string",
"term_vector":"with_positions_offsets",
"store":"yes"}
}
}
}
}
}'
and the following :
curl -X POST "http://localhost:9200/docs/attachment" -d #tmp.json
The problem is that the content is stored as it is in the file.
I was expecting the content to be decoded, like so :
base64.b64decode("IkdvZCBTYXZlIHRoZSBRdWVlbiIgKGFsdGVybmF0aXZlbHkgIkdvZCBTYXZlIHRoZSBLaW5nIg==")
That gives :
b'"God Save the Queen" (alternatively "God Save the King"'
To encode in base64, here what I do :
import json, base64
file64 = base64.b64encode(open('file.pdf', "rb").read()).decode('ascii')
f = open('tmp.json', 'w')
data = {"file":file64, "title":fname}
json.dump(data,f)
f.close()
I would like to be able to see the content using kibana (but for now I see only the base64 data ...)
This didn't work :
curl -X PUT "http://localhost:9200/docs/attachment/_mapping" -d '{
"attachment": {
"properties" : {
"content" : {
"type" : "attachment",
"fields" : {
"title" : {"store":"yes"},
"content":{
"type":"string",
"term_vector":"with_positions_offsets",
"store":"yes"}
}
}
}
}
}'
This worked, and I can see the content of the PDF through Kibana :
curl -X PUT "http://localhost:9200/docs" -d '{
"mappings" : {
"attachment" : {
"properties" : {
"content" : {
"type" : "attachment",
"fields" : {
"content" : { "store" : "yes" },
"author" : { "store" : "yes" },
"title" : { "store" : "yes"},
"date" : { "store" : "yes" },
"keywords" : { "store" : "yes", "analyzer" : "keyword" },
"name" : { "store" : "yes" },
"content_length" : { "store" : "yes" },
"content_type" : { "store" : "yes" }
}
}
}
}
}
}'
I'm trying to use the meteor-collectionapi package to update my database. I've set up a basic collection to test out the functionality.
I'm starting with this data:
{ "name" : "Darrell David", "age" : "18", "gender" : "Male", "_id" : "8BW9Yg2oKByBGdnSa" }
{ "name" : "Julie Smith", "age" : "21", "gender" : "Female", "_id" : "fAaFwCEXLzrmejnJK" }
{ "name" : "Todd Davis", "age" : "32", "gender" : "Male", "_id" : "ixKjhkTmjrNte2DjP" }
Now, I want to update the gender of the first player to "Female" so I call this using CURL:
curl -H "X-Auth-Token: 97f0ad9e24ca5e0408a269748d7fe0a0" -X PUT -d "{\"$set\":{\"gender\":\"Female\"}}" http://localhost:3000/collectionapi/players/8BW9Yg2oKByBGdnSa
And what I wind up with is this:
{ "_id" : "8BW9Yg2oKByBGdnSa", "" : { "gender" : "Female" } }
{ "name" : "Julie Smith", "age" : "21", "gender" : "Female", "_id" : "fAaFwCEXLzrmejnJK" }
{ "name" : "Todd Davis", "age" : "32", "gender" : "Male", "_id" : "ixKjhkTmjrNte2DjP" }
The first player has been completely overwritten and the name and age fields have been lost.
What am I missing here? When I execute this command in the MongoDB console it works perfectly:
db.players.update(
{ _id: "8BW9Yg2oKByBGdnSa" },
{ $set: { gender: "Female" } }
)
I'm guessing that bash is replacing "$set" with an empty environment variable
eg. echo "$set" vs echo "\$set"
so update your PUT command to:
curl -H "X-Auth-Token: 97f0ad9e24ca5e0408a269748d7fe0a0" -X PUT -d "{\"\$set\":{\"gender\":\"Female\"}}" http://localhost:3000/collectionapi/players/8BW9Yg2oKByBGdnSa
By default Collection.update() will replace a document if no modifiers are present ($set, $unset, $push, $pull etc). So the command being sent to the server is to replace the document with {"":{"gender":"Female"}}
when requesting data over Neo4j, as in, say,
curl -i -XPOST -d'{ "query" : "start n=node(*) return n" }'
-H "accept:application/json;stream=true"
-H content-type:application/json
http://localhost:7474/db/data/cypher
i get, as documented, a response like this:
{
"columns" : [ "n" ],
"data" : [ [ {
"outgoing_relationships" : "http://localhost:7474/db/data/node/0/relationships/out",
"data" : {
},
"traverse" : "http://localhost:7474/db/data/node/0/traverse/{returnType}",
"all_typed_relationships" : "http://localhost:7474/db/data/node/0/relationships/all/{-list|&|types}",
"property" : "http://localhost:7474/db/data/node/0/properties/{key}",
"self" : "http://localhost:7474/db/data/node/0",
"properties" : "http://localhost:7474/db/data/node/0/properties",
"outgoing_typed_relationships" : "http://localhost:7474/db/data/node/0/relationships/out/{-list|&|types}",
"incoming_relationships" : "http://localhost:7474/db/data/node/0/relationships/in",
"extensions" : {
},
"create_relationship" : "http://localhost:7474/db/data/node/0/relationships",
"paged_traverse" : "http://localhost:7474/db/data/node/0/paged/traverse/{returnType}{?pageSize,leaseTime}",
"all_relationships" : "http://localhost:7474/db/data/node/0/relationships/all",
"incoming_typed_relationships" : "http://localhost:7474/db/data/node/0/relationships/in/{-list|&|types}"
} ], [ {
"outgoing_relationships" : "http://localhost:7474/db/data/node/1/relationships/out",
"data" : {
"glyph" : "δΈ€",
"~isa" : "glyph"
},
"traverse" : "http://localhost:7474/db/data/node/1/traverse/{returnType}",
"all_typed_relationships" : "http://localhost:7474/db/data/node/1/relationships/all/{-list|&|types}",
"property" : "http://localhost:7474/db/data/node/1/properties/{key}",
"self" : "http://localhost:7474/db/data/node/1",
"properties" : "http://localhost:7474/db/data/node/1/properties",
"outgoing_typed_relationships" : "http://localhost:7474/db/data/node/1/relationships/out/{-list|&|types}",
"incoming_relationships" : "http://localhost:7474/db/data/node/1/relationships/in",
"extensions" : {
},
"create_relationship" : "http://localhost:7474/db/data/node/1/relationships",
"paged_traverse" : "http://localhost:7474/db/data/node/1/paged/traverse/{returnType}{?pageSize,leaseTime}",
"all_relationships" : "http://localhost:7474/db/data/node/1/relationships/all",
"incoming_typed_relationships" : "http://localhost:7474/db/data/node/1/relationships/in/{-list|&|types}"
} ], [ {
"outgoing_relationships" : "http://localhost:7474/db/data/node/2/relationships/out",
"data" : {
"~isa" : "LPG",
"LPG" : "1"
},
"traverse" : "http://localhost:7474/db/data/node/2/traverse/{returnType}",
"all_typed_relationships" : "http://localhost:7474/db/data/node/2/relationships/all/{-list|&|types}",
"property" : "http://localhost:7474/db/data/node/2/properties/{key}",
"self" : "http://localhost:7474/db/data/node/2",
"properties" : "http://localhost:7474/db/data/node/2/properties",
"outgoing_typed_relationships" : "http://localhost:7474/db/data/node/2/relationships/out/{-list|&|types}",
"incoming_relationships" : "http://localhost:7474/db/data/node/2/relationships/in",
"extensions" : {
},
"create_relationship" : "http://localhost:7474/db/data/node/2/relationships",
"paged_traverse" : "http://localhost:7474/db/data/node/2/paged/traverse/{returnType}{?pageSize,leaseTime}",
"all_relationships" : "http://localhost:7474/db/data/node/2/relationships/all",
"incoming_typed_relationships" : "http://localhost:7474/db/data/node/2/relationships/in/{-list|&|types}"
} ], [ {
and so on and on. the URLs delivered with each node are certainly well meant, but they also occupy a major portion of the data transmitted. they're also highly redundant and not what i', after with my query. is there any way to drop all of that traverse,
all_typed_relationships,
property,
self,
properties,
outgoing_typed_relationships,
incoming_relationships,
extensions,
create_relationship,
paged_traverse,
all_relationships,
incoming_typed_relationships
jazz?
The only way is to specify the properties you want returned in the return statement. Like:
return id(n), n.glyph;