I have setup Spark SQL on Jypterhub using Apache Toree SQL kernel. I wrote a Python function to update Spark configuration options in the kernel.json file for my team to change configuration based on their queries and cluster configuration. But I have to shutdown the running notebook and re-open or restart the kernel after running Python function. In this way, I'm forcing the Toree kernel to read the JSON file to pick up the new configuration.
I thought of implementing this shutdown and restart of kernel in a programmatic way. I got to know about the Jupyterhub REST API documentation and am able implement it by invoking related API's. But the problem is, the single user server API port is set randomly by the Spawner object of Jupyterhub and it keeps changing every time I spin up a cluster. I want this to be fixed before launching the Jupyterhub service.
Here is a solution I tried based on Jupyterhub docs:
sudo echo "c.Spawner.port = 35289
c.Spawner.ip = '127.0.0.1'" >> /etc/jupyterhub/jupyterhub_config.py
But this did not work as the port was again set by the Spawner randomly. I think there is a way to fix this. Any help on this would be greatly appreciated. Thanks
Related
I am trying to run a series of commands to configure a vlan on a Dell EMC OS10 server using Paramiko. However I am running into a rather frustrating problem.
I want to run the following
# configure terminal
(config)# interface vlan 3
(conf-if-vl-3)# description VLAN-TEST
(conf-if-vl-3)# end
However, I can't seem to figure out how to achieve this with paramiko.SSHClient().
When I try to use sshclient.exec_command("show vlan") it works great, it runs this command and exits. However, I don't know how to run more than one command with a single exec_command.
If I run sshclient.exec_command("configure") to access the configuration shell, the command completes and I believe the channel is closed, since my next command sshclient.exec_command("interface vlan ...") is not successful since the switch is no longer in configure mode.
If there is a way to establish a persistent channel with exec_command that would be ideal.
Instead I have resorted to a function as follows
chan = sshClient.invoke_shell()
chan.send("configure\n")
chan.send("interface vlan 3\n")
chan.send("description VLAN_TEST\n")
chan.send("end\n")
Oddly, this works when I run it from a Python terminal one command at a time.
However, when I call this function from my Python main, it fails. Perhaps the channel is closed too soon when it goes out of scope from the function call?
Please advise if there is a more reasonable way to do this
Regarding sending commands to the configure mode started with SSHClient.exec_commmand, see:
Execute (sub)commands in secondary shell/command on SSH server in Python Paramiko
Though it's quite common that "devices" do not support the "exec" channel at all:
Executing command using Paramiko exec_command on device is not working
Regarding your problem with invoke_shell, it's quite possible that the server needs some time to get ready for the next command.
Quick-and-dirty solution is to "sleep" shortly between the individual send calls.
Better solution to is to wait for command prompt before sending the next command.
I have installed the Hue on the Linux whixh is an instance from Azure. I have made all the required changes in ambari and hue.ini conf file. And when I run the supervisor job, it runs fine
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
hue 83933 sshuser 3u IPv4 15707246 0t0 TCP *:8000 (LISTEN)
But when I try to access the wb hue, I don't get any page loaded. It shows refused to connect.
Tried deleting caches and reset up was done.
I am using hue 4.7 version and I don't find any issues in error.log file. Yet, I don't see any data in access.log file. Could you please help me?
Do you have
http_host=0.0.0.0
in the hue.ini?
#Ruthikajawar here is a working hue.ini for ambari
https://github.com/steven-dfheinz/HDP3-Hue-Service/blob/Hue.4.6.0/configuration/live.hue.ini
I have noticed that sometimes, after initial install, it takes 1 or 2 restarts to get the WEBUI to work. I have also noticed sometimes, after a restart, it takes quite a few moments before the WEBUI starts to respond.
Give it some time after restart and check the WEBUI. If you still are not getting it to answer you need to check /var/log/hue/error.log as it should be very specific with errors causing the WEBUI to fail on startup.
look, always should run this command:
# cd /opt/openfire/bin/
# ./openfire.sh start
Openfire 3.6.4 [19-may-2009 17:43:25]
Console d'administration en écoute sur:
http://server.xxxxx.com:9090
https://server.xxxxx.com:9091
Index opened.
Index thread started
If I close the putty command line… the service openfire down… and after is neccessary again run… somebody know some way for always this service are permanently up ?
Thanks
All of the Openfire distributions should come with some kind of daemon or service script. It's hard to tell exactly what operating system you're on, but try typing something like this:
service openfire start
Also, I'm noticing that you're using a very old version of Openfire (it's from 2009!). I urge you to upgrade. Not only were many improvements and new features added, but security has been improved greatly over the years too.
I have a question about connecting Redis DB to Jmeter, using jp#gc - Redis Data Set.
I created a test and want to see a value from Redis, the problem is that When I add the Redis DB component nothing happened, I press the Play button and nothing happened.
I think I am not configured the Redis as expected in jmeter.
I didnt create any variable just name a new variable called dsos.
I just want to pass the value from redis of dsos_13_173 to the parameter dsos
1. How can I see why the configuration not succeed?
2. What I am missing?
I am using jmeter 3.2, with plugin v0.2 and installed it from plug in manager, and the DB is remote I am using IP not localhost as in all examples
Regards
Redis Data Set config acts alike CSV Data Set Config, so given you want to use the data from Redis in i.e. HTTP Request sampler you just need to refer to it as ${dsos} where required.
You can also double check the associated JMeter Variable value using Debug Sampler
See JMeter’s Redis Data Set - An Introduction for comprehensive explanation, step-by-step instruction and example test plan.
I have two clusters, one in local virtual machine another in remote cloud. Both clusters in Standalone mode.
My Environment:
Scala: 2.10.4
Spark: 1.5.1
JDK: 1.8.40
OS: CentOS Linux release 7.1.1503 (Core)
The local cluster:
Spark Master: spark://local1:7077
The remote cluster:
Spark Master: spark://remote1:7077
I want to finish this:
Write codes(just simple word-count) in IntelliJ IDEA locally(on my laptp), and set the Spark Master URL to spark://local1:7077 and spark://remote1:7077, then run my codes in IntelliJ IDEA. That is, I don't want to use spark-submit to submit a job.
But I got some problem:
When I use the local cluster, everything goes well. Run codes in IntelliJ IDEA or use spark-submit can submit job to cluster and can finish the job.
But When I use the remote cluster, I got a warning log:
TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
It is sufficient resources not sufficient memory!
And this log keep printing, no further actions. Both spark-submit and run codes in IntelliJ IDEA result the same.
I want to know:
Is it possible to submit codes from IntelliJ IDEA to remote cluster?
If it's OK, does it need configuration?
What are the possible reasons that can cause my problem?
How can I handle this problem?
Thanks a lot!
Update
There is a similar question here, but I think my scene is different. When I run my codes in IntelliJ IDEA, and set Spark Master to local virtual machine cluster, it works. But I got Initial job has not accepted any resources;... warning instead.
I want to know whether the security policy or fireworks can cause this?
Submitting code programatically (e.g. via SparkSubmit) is quite tricky. At the least there is a variety of environment settings and considerations -handled by the spark-submit script - that are quite difficult to replicate within a scala program. I am still uncertain of how to achieve it: and there have been a number of long running threads within the spark developer community on the topic.
My answer here is about a portion of your post: specifically the
TaskSchedulerImpl: Initial job has not accepted any resources; check
your cluster UI to ensure that workers are registered and have
sufficient resources
The reason is typically there were a mismatch on the requested memory and/or number of cores from your job versus what were available on the cluster. Possibly when submitting from IJ the
$SPARK_HOME/conf/spark-defaults.conf
were not properly matching the parameters required for your task on the existing cluster. You may need to update:
spark.driver.memory 4g
spark.executor.memory 8g
spark.executor.cores 8
You can check the spark ui on port 8080 to verify that the parameters you requested are actually available on the cluster.