url="http://spark-master:6066/v1/submissions/create"
headers = {'Content-Type': 'application/json'}
data1='''{"appResource": "file:pyton.py",
"sparkProperties": {
...
"spark.com.fortytools.servlet.BasicAuthFilter.param.user" : "****",
"spark.com.fortytools.servlet.BasicAuthFilter.param .password" : "****",
"spark.eventLog.enabled": "false",
"spark.app.name": "Spark_Test",
"spark.submit.deployMode": "CLIENT",
"spark.driver.supervise": "false"},
"clientSparkVersion": "3.2.0",
"mainClass": "org.apache.spark.deploy.SparkSubmit",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"},
"action": "CreateSubmissionRequest",
"appArgs": ["80"]}'''
I want to know why "client mode" is not working in spark standalone (The driver should not take resource if it is in client mode in worker node, but it does not, which means somehow client mode is not working.)
Related
Is there a way to automate user creation on GridGain Web Console's docker container deployment?
Our test stand deployment is fully automated, and we'd like to deploy Web Agent automatically as well, copying token and starting Agent's container manually every time is not very convenient in our case.
There are several options:
Create a Web Console user with HTTP REST API, grab their token and pass it to the Agent.
Generate your own token (a UUID), pass it to Agent, create a Web Console user with API calls and set their token.
Please keep in mind that the Web Console HTTP API is considered private. It has been stable for a while, especially the user-related parts, so I wouldn't expect any changes soon. Use it at your own discretion.
Before sending any requests, make sure you use a cookie jar. Send a "/api/v1/user" GET to initialize a session. The host is the same as WC, but you can also send requests to backend directly. CORS might be an issue.
In general, you can open browser network inspector, perform actions manually, note what requests are made and perform same requests with a tool of your choice, like curl. Some communications are handled by a Web Socket connection, but not for user management.
Endpoints you are interested in:
POST "/api/v1/user". Creates a user. Example payload:
{
"email": "user#example",
"password": "1",
"firstName": "User",
"lastName": "Name",
"phone": "+790000000",
"country": "Russia",
"company": "GridGain",
"industry": "Software"
}
POST "/api/v1/profile/save". Edits user. Example payload:
{
"firstName": "User",
"lastName": "Name",
"email": "test#example",
"phone": null,
"country": "Russia",
"company": "GridGain",
"industry": "Other",
"permitEmailContact": false,
"permitPhoneContact": false,
"token": "fcf99d68-5a4c-4a43-8abc-1f93e19af26a"
}
GET "/api/v1/user". Gets a user. Example payload:
{
"email": "test#example",
"firstName": "User",
"lastName": "name",
"phone": null,
"company": "GridGain",
"country": "Russia",
"admin": false,
"becomeUsed": false,
"industry": "Other",
"permitEmailContact": false,
"permitPhoneContact": false,
"token": "fcf99d68-5a4c-4a43-8abc-1f93e19af26a",
"lastEvent": 0
}
In order to receive Azure IotHub Device Twin change notifications, it appears that it's necessary to create a custom endpoint and create a route to send notifications to that endpoint. This seems straightforward enough on the Azure Portal, but as one might expect we want to automate it.
I haven't been able to find any documentation for the the az cli or even the REST API, though I might have missed something. I didn't find anything promising looking in the SDKs either.
How do I automate adding a custom endpoint and then setting up the route for device twin notifications?
You can check IotHubs template to see if it helps.
Route:
"routing": {
"endpoints": {
"serviceBusQueues": [
{
"connectionString": "string",
"name": "string",
"subscriptionId": "string",
"resourceGroup": "string"
}
]
},
"routes": [
{
"name": "string",
"source": "string",
"condition": "string",
"endpointNames": [
"string"
],
"isEnabled": boolean
}
],
Consumer group:
{
"apiVersion": "2016-02-03",
"type": "Microsoft.Devices/IotHubs/eventhubEndpoints/ConsumerGroups",
"name": "[concat(parameters('hubName'), '/events/cg1')]",
"dependsOn": [
"[concat('Microsoft.Devices/Iothubs/', parameters('hubName'))]"
]
},
For more detailed information you can reference:
Microsoft.Devices/IotHubs template reference
Create an IoT hub using Azure Resource Manager template (PowerShell)
I recently went through the tutorial for load balancing apps in DCOS using marathon-lb (in the example they balance some nginx containers: https://dcos.io/docs/1.9/networking/marathon-lb/marathon-lb-advanced-tutorial/). I am trying to use this approach to internally load balance my own custom application. The custom app I am using is a play scala app. I have the internal marathon-lb set up and can successfully use it for the nginx container but when I try to use my own docker image I cannot get this to work. I start up my service with my custom image and I can access the service fine by using the IP and port that gets assigned to it (i.e. if the service gets deployed on 10.0.0.0 and is available on port 1234 then curl http://10.0.0.0:1234/ works as expected and I can also make my api calls as defined in my application routes). However, when I try to access the app through the load balancer (curl -i http://marathon-lb-internal.marathon.mesos:10002, where 10002 is the service port) then I get this message:
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
For reference, here is my json file I'm using to start my custom service:
{
"id": "my-app",
"container": {
"type": "DOCKER",
"docker": {
"image": "my_repo/my_image:1.0.0",
"network": "BRIDGE",
"portMappings": [
{ "hostPort": 0, "containerPort": 9000, "servicePort": 10002, "protocol": "tcp" }
],
"parameters": [
{ "key": "env", "value": "USER_NAME=user" },
{ "key": "env", "value": "USER_PASSWORD=password" }
],
"forcePullImage": true
}
},
"instances": 1,
"cpus": 1,
"mem": 1000,
"healthChecks": [{
"protocol": "HTTP",
"path": "/v1/health",
"portIndex": 0,
"timeoutSeconds": 10,
"gracePeriodSeconds": 10,
"intervalSeconds": 2,
"maxConsecutiveFailures": 10
}],
"labels":{
"HAPROXY_GROUP":"internal"
},
"uris": [ "https://s3.amazonaws.com/my_bucket/my_docker_credentials" ]
}
I had the same problem and found the solution here
marathon-lb health check failing on all spray.io containers
Need to add
"HAPROXY_0_BACKEND_HTTP_HEALTHCHECK_OPTIONS": " http-send-name-header Host\n timeout check {healthCheckTimeoutSeconds}s\n"
To your config so that the REST layer doesn't bark on the health check from marathon
We are using Camunda for our approval process implementation in our application.
We created a BPMN process with human Task service. We are using the below URL
engine-rest/engine/default/process-definition/key/processKey/start
we pass our form parameters as input to this service
{
"variables": {
"requestId" : {"value" : "xxxxx", "type" : "String"},
"catalog" : {"value" : "yyyy", "type" : "String"},
"businessReason": {"value":"yyyyy","type":"String"},
"link": {"value":"","type":"String"}
}
}
The response of this start task is below-
{
"links": [
{
"method": "GET",
"href": "http://localhost:8080/engine-rest/engine/default/process-instance/31701",
"rel": "self"
}
],
"id": "31701",
"definitionId": "xxxxx:7:31605",
"businessKey": null,
"caseInstanceId": null,
"ended": false,
"suspended": false,
"tenantId": null
}
The id in the response is not the actual task ID which we use to get the task details etc instead its the execution ID.
Is there a way to get the task id back in the response.? Also can we add some parameteres to the above response. Like
"status" : "success"
I am having listener class created for the Human task but not sure how to add response parameters . Any help is appreciated
This is not possible unless you build a custom REST resource on top of Camunda's Java API. See https://docs.camunda.org/manual/7.6/reference/rest/overview/embeddability/ for info how you would embed the default REST resources into a custom JAX-RS application.
I am developing one video calling application. Currently using Xirsys's stun and turn server. I am using the result of https://service.xirsys.com/ice as my configurations. Is it the right username and credential to use in the Javascript page or anything else. If it is wrong then please guide me where will I get the correct iceServers values.
iceServers = [
{ "url": "stun:turn01.uswest.xirsys.com" },
{ "username": "0xxxxxx8-fxxc-1xx6-bxxb-bxxxxxxxxxx8", "url": "turn:turn01.uswest.xirsys.com:80?transport=udp", "credential": "0xxxxxxe-fxxc-1xx6-axx0-axxxxxxxxxx9" },
{ "username": "0xxxxxx8-fxxc-1xx6-bxxb-bxxxxxxxxxx8", "url": "turn:turn01.uswest.xirsys.com:3478?transport=udp", "credential": "0xxxxxxe-fxxc-1xx6-axx0-axxxxxxxxxx9" },
{ "username": "0xxxxxx8-fxxc-1xx6-bxxb-bxxxxxxxxxx8", "url": "turn:turn01.uswest.xirsys.com:80?transport=tcp", "credential": "0xxxxxxe-fxxc-1xx6-axx0-axxxxxxxxxx9" },
{ "username": "0xxxxxx8-fxxc-1xx6-bxxb-bxxxxxxxxxx8", "url": "turn:turn01.uswest.xirsys.com:3478?transport=tcp", "credential": "0xxxxxxe-fxxc-1xx6-axx0-axxxxxxxxxx9" },
{ "username": "0xxxxxx8-fxxc-1xx6-bxxb-bxxxxxxxxxx8", "url": "turns:turn01.uswest.xirsys.com:443?transport=tcp", "credential": "0xxxxxxe-fxxc-1xx6-axx0-axxxxxxxxxx9" },
{ "username": "0xxxxxx8-fxxc-1xx6-bxxb-bxxxxxxxxxx8", "url": "turns:turn01.uswest.xirsys.com:5349?transport=tcp", "credential": "0xxxxxxe-fxxc-1xx6-axx0-axxxxxxxxxx9" }
];
Note: Its working in the same network but not in different network. Even in different network I can get the incomming call but after receiving the call iceConnectionState gets failed.
I have also raised similar question here where I was using numb as stun and turn server.
Thanks in advance.
The ICE string should be used 'as-is' in the ice configuration for your WebRTC application. Note, however, that the ICE credentials are only valid for 30 seconds. You need to request a fresh ICE string immediately before each connection.
Local network connections will work WITHOUT a valid ICE string, because your NAT translation will NOT use TURN / STUN. This is because your NAT translates your local IP's itself. Therefore, local network connections will always work (unless you have a non-common NAT situation). If you were using the ICE credentials without refreshing them before each call, that will certainly be why your external connections were failing (or certainly will contribute to the problem).
Lee