getting this error during startup time of scylla node as I am loading data
Nov 12 21:55:13 usw1-im-stage-scylladb1 scylla[53703]: [shard 0] database - Keyspace product_prod: Reading CF cleanup_transaction id=bb0a0640-058f-11ea-b8e4-00000000000c version=dde3ee6f-185b-37ba-80fb-6425cce4532f
Nov 12 22:10:02 usw1-im-stage-scylladb1 systemd[1]: scylla-server.service start operation timed out. Terminating.
running this on scylla enterprise 2019.1.2
Scylla's documentation includes KBs and FAQ. I think this KB is spot-on, what you are looking for:
https://docs.scylladb.com/troubleshooting/scylla_wont_start/#solution
Here is the solution suggested there:
Locate the directory with the systemd files where the scylla-server.service resides.
For Centos operating systems it is expected to be under /usr/lib/systemd/system/scylla-server.service
For Ubuntu operating systems it is expected to be under /etc/systemd/system/scylla-server.service.d
Create the following directory (if not exist)
Centos
sudo mkdir /usr/lib/systemd/system/scylla-server.service
Ubuntu
sudo mkdir /etc/systemd/system/scylla-server.service.d
Create a file inside that directory named 10-timeout.conf, with the following contents:
[Service]
TimeoutStartSec=9000
Reload the systemd Daemon for the new configurations to take in effect.
systemctl daemon-reload
Related
So I'm not sure how to even figure out what's happening. Running redis-server version 6.2.5 on RHEL 7.9. I can start REDIS with the user I want to run it as from the command line. But when I try to start it as a systemd service with the REDIS.service file:
[Unit]
Description=REDIS Service
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
Type=simple
User=myuser
Group=myuser
WorkingDirectory=/app/redis
ExecStart=/bin/redis-server /app/redis/primary.conf
Restart=on-failure
[Install]
WantedBy=multi-user.target
It fails. The journalctl entry shows:
Aug 27 08:57:18 computerName redis-server[119338]: 119338:C 27 Aug 2021
08:57:18.981 # Fatal error, can't open config file
'/app/redis/primary.conf': Permission denied
The redis directory and all of it's files are owned by the user I'm telling to run the service. I even went to the extent of putting 777 on the directory and files. Thanks for any help you can give me!
Eric
This server was a mess. There was previous install of redis on the server, the rpm database was corrupt, there were old services that weren't running but were still there. I had to do a lot of cleanup and perform a clean install and it started working.
I'm trying to learn how to backup and restore my Ubuntu Server via tar so I know that I have a safe system. After I untar and reboot, I have several issues, but they seem to be caused by a read-only file system. The source and destination server are both Ubuntu Server on the same version, 18.04.05 LTS. The source server is a VPS that has 6 GB RAM and 4vCPUs. The destination server is a VM on my FreeNAS machine with 6GB RAM and 2 vCPUs.
The primary applications that need to work are my Graylog server and Nagios server. I've mostly followed the instructions at Ubuntu.
First, my tar command is:
sudo tar -c --use-compress-program=pigz -f backup.tar.gz --exclude=/backup.tar.gz --exclude=/dev --exclude=/usr --exclude=/sbin --exclude=/proc --exclude=/sys --exclude=/tmp --exclude=/run --exclude=/mnt --exclude=/media --exclude=/lost+found --exclude=/home/*/.cache --exclude=/home/*/.gvfs --exclude=/home/*/.local/share/Trash --exclude=/var/log --exclude=/var/cache/apt/archives --exclude=/usr/src/linux-headers* --one-file-system /
I use pigz to utilize the VPS's 4 vCPUs to take less time. I transfer this to my VM which as a fresh copy of Ubuntu Server 18.04.05 and untar with:
sudo tar -xvpzf backup.tar.gz -C / --numeric-owner
After I reboot, I get the following as soon as I boot:
Unable to setup logging. [Errno 30] Read-only file system: '/var/log/landscape/sysinfo.log'
run-parts: /etc/update-motd.d/50-lanscape-sysinfo exited with return code 1
mktemp: failed to create file via template '/var/lib/update-notifier/tmp.XXXXXXXXXX': Read-only file system
run-parts: /etc/update-motd.d/95-hwe-eol exited with return code 1
/usr/lib/update-notifier/update-motd-fsck-at-reboot: 33: /usr/lib/update-motd-fsck-at-reboot: cannot create /var/lib/update-notifier/fsck-at-reboot: Read-only file system
I do see that some areas of the system do work like the original source. My SSH port changes, hostname changes, etc. But I get these above errors and my Graylog and Nagios servers do not work.
So I'm wondering where I went wrong in my process and any help would be appreciated. The source is a live server with backups so I'm safe there. I'm just making sure I have my ducks in a row for the future.
I have configured a systemd Unit for DataStax Enterprise 4.8.5:
### /etc/systemd/system/dse1.service
[Unit]
Description=DataStax Enterprise
[Service]
User=cassandra
ExecStart=/opt/dse/dse1/bin/dse cassandra -k
ExecStop=/opt/dse/dse1/bin/dse cassandra-stop
when I execute sudo systemctl start dse1, if i immediately do a status afterwards, i get:
● dse1.service - DataStax Enterprise 1
Loaded: loaded (/etc/systemd/system/dse1.service; static; vendor preset: disabled)
Active: active (running) since Wed 2016-03-23 13:47:57 EDT; 1s ago
Main PID: 31699 (cassandra)
CGroup: /system.slice/dse1.service
├─31699 /bin/sh /opt/dse/dse1/resources/cassandra/bin/cassandra -k -Djava.library.path=:/opt/dse/dse1/resources/hadoop/native...
├─31894 /bin/java -cp :/opt/dse/dse1/lib/dse-core-4.8.5.jar:/opt/dse/dse1/lib/dse-hadoop-4.8.5.jar:/opt/dse/dse1/lib/dse-hive...
└─31895 grep -q Error: Exception thrown by the agent : java.lang.NullPointerException
If I then wait a few seconds and try again, I get:
● dse1.service - DataStax Enterprise 1
Loaded: loaded (/etc/systemd/system/dse1.service; static; vendor preset: disabled)
Active: inactive (dead)
Mar 23 13:34:28 pspldsea01p.fleet.ad systemd[1]: Started DataStax Enterprise 1.
Mar 23 13:34:28 pspldsea01p.fleet.ad systemd[1]: Starting DataStax Enterprise 1...
Mar 23 13:38:33 pspldsea01p.fleet.ad systemd[1]: Started DataStax Enterprise 1.
Mar 23 13:38:33 pspldsea01p.fleet.ad systemd[1]: Starting DataStax Enterprise 1...
Mar 23 13:47:41 pspldsea01p.fleet.ad systemd[1]: Started DataStax Enterprise 1.
Mar 23 13:47:41 pspldsea01p.fleet.ad systemd[1]: Starting DataStax Enterprise 1...
Mar 23 13:47:44 pspldsea01p.fleet.ad dse[31267]: nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
Mar 23 13:47:57 pspldsea01p.fleet.ad systemd[1]: Started DataStax Enterprise 1.
Mar 23 13:47:57 pspldsea01p.fleet.ad systemd[1]: Starting DataStax Enterprise 1...
Mar 23 13:48:01 pspldsea01p.fleet.ad dse[32004]: nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
Hint: Some lines were ellipsized, use -l to show in full.
If I just execute /opt/dse/dse1/bin/dse cassandra -k as the cassandra user, it works fine.
I can't seem to find any additional logging in the normal logging locations or with sudo journalctl -u dse1
Any ideas? Thanks!
It is unfortunate that DataStax Enterprise doesn't come with a systemd service file to be able to use systemctl. However, it does come with an init script. Full documentation is available at the docs
Basically you have two options. The first one is to use the init.d directly, by starting the service:
sudo service dse start
I'm , however, too used to systemctl now to go back to that. So this is my systemd service file
[Unit]
Description=DataStax Enterprise
After=network.target
[Service]
PIDFile=/var/run/dse/dse.pid
ExecStart=/etc/init.d/dse start
ExecStop=/etc/init.d/dse stop
SuccessExitStatus=143
TimeoutSec=300
[Install]
WantedBy=multi-user.target
The init script has many configuration options. For the sake of simplicity, it may be wise to just use those directly in the script. For example, you specify the user in your systemd service file. That was giving me problems until I noticed that the user is already specified in the script. No need to duplicate options.
The SucessExitStatus=143 option is a common configuration for Java applications.
You may have to adapt the location of the script if you didn't install DSE with your package manager
Adding this in case post late but still if this can be useful.
Adding with adding additional parameter RemainAfterExit
/etc/systemd/system/dse1.service
[Unit]
Description=DataStax Enterprise
[Service]
User=cassandra
RemainAfterExit=yes
ExecStart=/opt/dse/dse1/bin/dse cassandra -k
ExecStop=/opt/dse/dse1/bin/dse cassandra-stop
[Install]
WantedBy=multi-user.target
So after restarting my httpd redis crashed (due to the number of sudden requests sent via httpd and written on redis) and now when I try to restart redis on my centos 6.5 server I get the following error:
[root#host /]# /usr/sbin/redis-server restart
[1705] 17 Apr 00:30:49 # Fatal error, can't open config file 'restart'
I have also tried to login to redis using redis-cli and I get an error stating the connection to the server failed.
What options do I have to safely restart the server?
From the /src directory where you downloaded and unzipped your redis source, run the following. This is for RHEL based systems.
make install
# (OR)
sudo cp src/redis-server /usr/local/bin/
sudo cp src/redis-cli /usr/local/bin/
I'm trying to run activemq on my ubuntu virtual machine but have constantly been running into issues getting it to start up. I've tried downloading the binary and source with no luck. Currently I have downloaded the source, run "mvn clean install -Dmaven.test.skip=true" and mvn reported successful installation. I then hunted around in my .m2 folder found apache-activemq-5.5.1-bin.tar.gz and extracted it to my home/USERNAME dir and attempted to run "bash bin/activemq start" only to receive the following error.
INFO: Loading '/etc/default/activemq'
INFO: Using java '/usr/bin/java'
INFO: Starting - inspect logfiles specified in logging.properties
and log4j.properties
to get details
bin/activemq: line 370: /usr/bin/java -Xms256M -Xmx256M -Dorg.apache.activemq.UseDedicatedTaskRunner=true
-Djava.util.logging.config.file=logging.properties
-Dcom.sun.management.jmxremote
-Dactivemq.classpath="/home/jacob/activeMq1/apache-activemq-5.5.1/conf;"
-Dactivemq.home="/home/jacob/activeMq1/apache-activemq-5.5.1"
-Dactivemq.base="/home/jacob/activeMq1/apache-activemq-5.5.1"
-jar "/home/jacob/activeMq1/apache-activemq-5.5.1/bin/run.jar" start >/dev/null 2>&1 &
RET="$?"; APID="$!";
echo $APID > /home/jacob/activeMq1/apache-activemq-5.5.1/data/activemq.pid;
echo "INFO: pidfile created : '/home/jacob/activeMq1/apache-activemq-5.5.1/data/activemq.pid' (pid '$APID')";
exit $RET: No such file or directory
Has any one run into this type of error before?
Looks like I'm answering one of my questions again, but maybe this will help someone in the future.
steps.
I ended up getting activemq to work by creating a configuration file via running the command "./bin/activemq setup newConfig" (exclude the quotes)
I then replaced the current config file "activemq" which was located at etc/default/. (I made a backup of the original activemq file before overwriting it with newConfig).
Run "./bin/activemq start" which will create a PID file.
After the file is created re-run "./bin/activemq start" to finally start up the broker.
You can then test the install by navigating to "http://localhost:8161/admin/" or by doing a "netstat -an | grep 61616" if you kept the default ports etc.
I install activemq 5.13 on Debian, download and unzip in /opt, then I go to /opt/apache-activemq-5.13.1/run "./bin/activemq start", then appear this error:
xx#debian:/opt/apache-activemq-5.13.1$ ./bin/activemq start
INFO: Loading '/etc/default/activemq'
INFO: Using java '/usr/bin/java'
INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details
./bin/activemq: 330: ./bin/activemq: "/usr/bin/java" -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/apache-activemq-5.13.1//conf/login.config -Djava.awt.headless=true -Djava.io.tmpdir="/opt/apache-activemq-5.13.1//tmp" -Dactivemq.classpath="/opt/apache-activemq-5.13.1//conf:/opt/apache-activemq-5.13.1//../lib/:" -Dactivemq.home="/opt/apache-activemq-5.13.1/" -Dactivemq.base="/opt/apache-activemq-5.13.1/" -Dactivemq.conf="/opt/apache-activemq-5.13.1//conf" -Dactivemq.data="/opt/apache-activemq-5.13.1//data" -jar "/opt/apache-activemq-5.13.1//bin/activemq.jar" start >/dev/null 2>&1 &
RET="$?"; APID="$!";
echo $APID > /opt/apache-activemq-5.13.1//data/activemq.pid;
echo "INFO: pidfile created : '/opt/apache-activemq-5.13.1//data/activemq.pid' (pid '$APID')";exit $RET: not found
What I did is to check the Debian version using "uname -a":
Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux
I found my debian is 64 bit system. so I run
./bin/linux-x86-64/activemq start
It shows:
Starting ActiveMQ Broker...
Then I can access the site:http://localhost:8161/admin/ with username "admin" and password "admin"
with Ubuntu 14.04, I had to create a link in /etc/activemq/instances-enabled
sudo ln -s ../instances-available/main/
similar to apache2 setup
then started the server with /etc/init.d/activemq start
sudo is necessary.
bin$ sudo ./activemq start
bin$ sudo ./activemq status
INFO: Loading '/opt/runtime/apache-activemq-5.11.1/bin/env' INFO: Using java '/usr/bin/java' ActiveMQ is running (pid '29887')