In aws emr, after modified the config file in /etc/presto/conf, how can we restart presto-server? Just on master node or on all nodes?
On EMR you can restart Presto with
sudo stop presto
sudo start presto
You should do this on every node where you modified the config file. You should also update the config file on every node, as appropriate.
Related
Background
Yesterday our machine crashed unexpectedly and our AOF file for Redis got corrupted.
Upon trying to start the service with sudo systemctl start redis-server we are greeted with the following logs:
Bad file format reading the append only file: make a backup of your
AOF file, then use ./redis-check-aof --fix
Research
Aparently this looks like a simple error to fix, just execute ./redis-check-aof --fix <filename>.
Except I don't have the smallest idea of where that file is.
I have searched the Github discussions for this issue, but unfortunately none provides me with the location for the file:
https://github.com/antirez/redis/issues/4931
The persistence documentation also doesn't make a mention of the location for this file:
https://redis.io/topics/persistence
Specs
These are the specs of the system where I am running Redis:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
Question
Where is located this file?
You have two choices:
Find the configure file for Redis, normally, it's named redis.conf. The dir and appendfilename configuration specify the directory and file name of the AOF file.
Connect to Redis with redis-cli, and use the CONFIG GET command to get the dir configuration, i.e. CONFIG GET dir. The AOF file should located under this directory.
The path is typically /var/lib/redis/appendonly.aof you will need to run sudo redis-check-aof --fix /var/lib/redis/appendonly.aof
in case if you use docker and append volume to /data then the path to appendonly.aof will be: /data/appendonly.aof
In my case, I was using docker. I started the redis server without using --appendonly yes, then it started without any issues. And then ran CONFIG GET dir like #for-stack said and got this output:
1) "dir"
2) "/data"
So I checked under the /data path and found the file appendonly.aof
Then I ran /usr/local/bin/redis-check-aof --fix /data/appendonly.aof to fix the issue.
I ran /path/redis-check-aof --fix /data/appendonly.aof to fix this.
Thanks all.
I can not a start a yarn session of Apache Flink in Amazons EMR. The error message I get is
$ tar xvfj flink-0.9.0-bin-hadoop26.tgz
$ cd flink-0.9.0
$ ./bin/yarn-session.sh -n 4 -jm 1024 -tm 4096
...
Diagnostics: File file:/home/hadoop/.flink/application_1439466798234_0008/flink-conf.yaml does not exist
java.io.FileNotFoundException: File file:/home/hadoop/.flink/application_1439466798234_0008/flink-conf.yaml does not exist
...
I am using Flink verision 0.9 and Amazons Hadoop version 4.0.0. Any ideas or hints?
The full log can be found here: https://gist.github.com/headmyshoulder/48279f06c1850c62c28c
From the log:
The file system scheme is 'file'. This indicates that the specified Hadoop configuration path is wrong and the sytem is using the default Hadoop configuration values.The Flink YARN client needs to store its files in a distributed file system
Flink failed to read the Hadoop configuration files. They are either picked up from the environment variables, e.g. HADOOP_HOME, or you can set the configuration dir in the flink-conf.yaml before you execute your YARN command.
Flink needs to read the Hadoop configuration to know how to upload the Flink jar to the cluster file system such that the newly created YARN cluster can access it. If Flink fails to resolve the Hadoop configuration, it uses the local file system for uploading the jar. That means that the jar will be put on the machine you launch your cluster from. Thus, it won't be accessible from the Flink YARN cluster.
Please see the Flink configuration page for more information.
edit: On Amazong EMR, export HADOOP_CONF_DIR=/etc/hadoop/conf let's Flink discover the Hadoop configuration directory.
if i were you i would try with this:
./bin/yarn-session.sh -n 1 -jm 768 -tm 768
Redis went quite on me.
user#mycomputer:~$ redis-cli
Could not connect to Redis at 127.0.0.1:6379: Connection refused
I try to restart the service by doing this
sudo /etc/init.d/redis_6379 stop
/var/run/redis/redis.pid exists, process is already running or crashed
But no luck. Logs didn't show an error as well.
Got it fixed by backing up the redis.rdp file mine is located at
/var/lib/redis
check your config file "/etc/redis/redis.conf" for the rdp file's location and do this
sudo mv /var/lib/redis/redis.rdp /var/lib/redis/redis_backup.rdp
Then recreate the the redis.rdp file
sudo touch redis.rdp
Run the redis-server with the conf and it should work
sudo redis-server /etc/redis/redis.conf
Get it fixed in a tidy way: Recreate the the redis.rdp file as suggested here in one of answer, will purge all the cache recorded so far and redis will start up fresh with no cache data.
This is a warning message to notify system crash / improper shutdown: "/var/run/redis/redis.pid exists, process is already running or crashed"
Just delete /var/run/redis/redis.pid file and restart the server again.
Note: You might have lost latest cache changes due to untidy shutdown, which weren't flushed into the disk. This data loss can be minimized using frequent disk flush configuration in redis conf file(in my case it is #/etc/redis/6379.conf)
save 900 1
save 300 10
save 60 10000
Or try AOF persistence, more details [here][1]
Depends on how you installed redis, the pid can be found on /var/run/redis_6379.pid.
What happened is that redis crashed, but the pid is still there. So you just have to delete it.
sudo rm -f /var/run/redis_6379.pid
Then start redis again:
sudo /etc/init.d/redis_6379 start
If you can't find it, I suggest installing redis "more properly". Follow redis quickstart guide in the Installing Redis more properly section.
You can find it here:
https://redis.io/topics/quickstart
Run the redis-server with config.
sudo redis-server redis.conf
I am using hadoop 1.2.1 having 3 data nodes and one namenode. My hbase version is 0.94.14. I have configured apache hive 1.0 on name node machine.
I have to import hbase table data to hive. When I run a query, it gives following error in log file
ERROR org.apache.hadoop.hbase.mapreduce.TableInputFormatBase - Cannot resolve the host name for /192.168.3.9 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '9.3.168.192.in-addr.arpa'
What is the problem in my setup. I have followed this tutorial for hadoop installation.
In hadoop namenode log file following warning appears when I run query in hive
WARN org.apache.hadoop.hdfs.server.namenode.FSEditLog: Cannot roll edit log, edits.new files already exists in all healthy directories:
Is there any information needed for hive about how many datanode hadoop have?
Also my Hmaster is running on some other machine and I have configured hive at namnode machine/
Your hadoop, zookeeper, hbase and hive should be in running condition.
1) COPY THESE FILES TO THE HADOOP LIBRARY.
sudo cp /usr/lib/hive/lib/hive-common-0.7.0-cdh3u0.jar /usr/lib/hadoop/lib/
sudo cp /usr/lib/hive/lib/hbase-0.90.1-cdh3u0.jar /usr/lib/hadoop/lib/
2)STOP HBASE AND HADOOP USING FOLLOWING COMMONDS
/usr/lib/hadoop/bin/stop-all.sh
/usr/lib/hbase/bin/stop-hbase.sh
3) RESTART HBASE AND HADOOP USING COMMONDS
/usr/lib/hadoop/bin/start-all.sh
/usr/lib/hadoop/bin/start-hbase.sh
I can't seem to automatically bootup my celeryd script located in /etc/init.d/celeryd everytime my Amazon Linux AMI 2013.03.1 machine is booted. I have to automatically do /etc/init.d/celeryd start . However, it boots perfectly and work right away.
Any ideas? I tried
sudo chkconfig /etc/init.d/celeryd on
You need to write a simple startup script:
Create a file called celeryd.sh
vim /etc/init.d/celeryd.sh
Inside that file:
#!/bin/sh
##Starts the celery on boot up###
/etc/init.d/celeryd start
Change permission:
chmod +x celeryd.sh
Done.
You can do init 6 and test, if it works or not.
More on : http://www.cyberciti.biz/tips/linux-how-to-run-a-command-when-boots-up.html