I've setup gitolite with shell access, and using Capistrano to deploy my code to production. The problem is that Capistrano bundles multiple commands in one line, using newlines and gitolite has a security check that looks for newlines, and dies. I'm not sure whether to tackle this from the Capistrano or Gitolite side.
I'm seeing this error from running 'cap deploy'
executing "rm -rf /home/git/public_html/project/releases/20101129165633/log
/home/git/public_html/project/releases/20101129165633/public/system
/home/git/public_html/project/releases/20101129165633/tmp/pids &&\\\n
mkdir -p /home/git/public_html/project/releases/20101129165633/public &&\\\n
mkdir -p /home/git/public_html/project/releases/20101129165633/tmp &&\\\n ln -s /home/git/public_html/project/shared/log /home/git/public_html/project/releases/20101129165633/log &&\\\n ln -s /home/git/public_html/project/shared/system /home/git/public_html/project/releases/20101129165633/public/system &&\\\n ln -s /home/git/public_html/project/shared/pids /home/git/public_html/project/releases/20101129165633/tmp/pids"
servers: ["projectsite.com"]
[projectsite.com] executing command
ERROR MESSAGE:
** [out :: projectsite.com] I don't like newlines in the command: <COMMAND FROM ABOVE>
The gitolite code that handles this is here: https://github.com/sitaramc/gitolite/blob/pu/src/gl-auth-command
You've probably figured this out by now, but not seeing an answer to this made me sad.
Instead of newlines, you can join multiple commands with "; ". Here is an example deploy script:
role :server, "projectsite.com"
namespace :deploy do
desc "Does whatever beeudoublez wants"
task :default, :roles => :server, :except => { :no_release => true } do
run [ "rm -rf /home/git/public_html/project/releases/20101129165633/log /home/git/public_html/project/releases/20101129165633/public/system /home/git/public_html/project/releases/20101129165633/tmp/pids",
"mkdir -p /home/git/public_html/project/releases/20101129165633/public",
"mkdir -p /home/git/public_html/project/releases/20101129165633/public"].join("; ")
end
end
Related
Using bitbucket pipelines to push to our remote from the build process that you get from the pipeline.
This is a snippet of the bitbucket-pipelines.yml file
- pipe: atlassian/ssh-run:0.2.2
variables:
SSH_USER: $PRODUCTION_USER
SERVER: $PRODUCTION_SERVER
COMMAND: '''rsync -zrSlh -e "ssh -p 22007" --stats --max-delete=0 $BITBUCKET_CLONE_DIR/ $PRODUCTION_USER#$PRODUCTION_SERVER:home/$PRODUCTION_USER'''
PORT: '22007'
The connection itself works, and it does run the command correctly once it is remoted onto the server...
INFO: Executing the pipe...
INFO: Using default ssh key
INFO: Executing command on {HOST}
ssh -A -tt -i /root/.ssh/pipelines_id -o StrictHostKeyChecking=no -p 22007 {USER}#{HOST} 'rsync -zrSlh -e "ssh -p 22007" --stats --max-delete=0 /opt/atlassian/pipelines/agent/build/ {USER}#{HOST}:home/{USER}'
bash: rsync -zrSlh -e "ssh -p 22007" --stats --max-delete=0 /opt/atlassian/pipelines/agent/build/ {USER}#{HOST}:home/{USER}: No such file or directory
Connection to {HOST} closed.
I've tried to run the same command locally from the directory on my machine
ssh -A -tt -i /root/.ssh/pipelines_id -o StrictHostKeyChecking=no -p 22007 {USER}#{HOST} 'rsync -zrSlh -e "ssh -p 22007" --stats --max-delete=0 "$PWD" {USER}#{HOST}:/home/{USER}'
but it just duplicates the home directory on the remote.
It looks to me like it's looking for the source directory on the server and not looking at the docker container from bitbucket (or the files on my local machine with pwd).
If I try to run the command without the '' then it fails because it's using port 22 by default. I've also tried offsetting the command into a bash script and using MODE: 'Script' which is an acceptable pattern for the plugin, but I can't use my environment variables in the sh file.
If all you wan't to do is to copy the files from the pipeline to the production server, you should you the rsync-deploy pipe, instead of the ssh-run. Your pipe configuration is gonna look pretty much like the following:
script:
- pipe: atlassian/rsync-deploy:0.3.2
variables:
USER: $PRODUCTION_USER
SERVER: $PRODUCTION_USER
REMOTE_PATH: 'home/$PRODUCTION_USER'
LOCAL_PATH: 'build'
SSH_PORT: '22007'
Make sure to configure your SSH keys in pipelines properly (here is a link to our docs for configuring SSH keys https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html)
I've found another way around this instead of needing a plugin, instead I'm running an rsync as a script step
image: atlassian/default-image:latest
- rsync -rltDvzCh --max-delete=0 --stats --exclude-from=excludes -e 'ssh -e none -p 22007' $BITBUCKET_CLONE_DIR/ $PRODUCTION_USER#$PRODUCTION_SERVER:/home/$PRODUCTION_USER
It seems the -e none is an important addition, as is loading in the atlassian image, as fails to find the rsync function, otherwise. I found this info on this post on Atlassian Community.
This seems to work pretty well for me
image: node:10.15.3
pipelines:
default:
- step:
name: <project-path>
script:
- apt-get update && apt-get install -y rsync
- ssh-keyscan -H $SSH_HOST >> ~/.ssh/known_hosts
- cd $BITBUCKET_CLONE_DIR
- rsync -r -v -e ssh . $SSH_USER#$SSH_HOST:/<project-path>
- ssh $SSH_USER#$SSH_HOST 'cd <project-path> && npm install'
- ssh $SSH_USER#$SSH_HOST 'pm2 restart 0'
Note: Avoid using sudo cmd in pipeline scripts
same issue with atlassian/default-image:3
rsync -azv ./project_path/*
bash: rsync: command not found
Solution:
apt-get update && apt-get install -y rsync
I know that this problem has been asked many times, but I can't get it sorted (I'm a beginner).
What I'm trying to do is to deploy my rails application to my production server using capistrano. I stored my project on a directory on gitlab. Everything was working perfectly until I moved my application in an other gitlab repository (git#gitlab.com:myusername/xxxxxx.git).
I think I set up my deploy.rb file accordingly :
set :application, "xxxxxx"
set :user, "yyyyy"
set :repository, "git#gitlab.com:myusername/xxxxxx.git"
But when I try to deploy it, I get the permission error :
[xxxxxx.com] executing command
[xxxxxx.com] env PATH=/home/kar/.rbenv/shims:/home/kar/.rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin sh -c 'if [ -d /var/www/xxxxxx/shared/cached-copy ]; then cd /var/www/xxxxxx/shared/cached-copy && git fetch -q origin && git fetch --tags -q origin && git reset -q --hard 97ff4f45240a680c1d278325d7ac1871536c8091 && git clean -q -d -x -f; else git clone -q git#gitlab.com:myusername/xxxxxx.git /var/www/xxxxxx/shared/cached-copy && cd /var/www/xxxxxx/shared/cached-copy && git checkout -q -b deploy 97ff4f45240a680c1d278325d7ac1871536c8091; fi'
** [xxxxxx.com :: err] Permission denied (publickey).
** [xxxxxx.com :: err] fatal: The remote end hung up unexpectedly
Could you please propose me some tests to find out from where the issue comes ?
Is there any key to add on my server ?
Thanks a lot for your help.
Here's the capistrano 3 plugin that is created solely for the purpose of troubleshooting problems like this one: capistrano-ssh-doctor.
The plugin outputs a report with found issues and suggested next steps.
I want to figure out how to set the path to ssh in git-ssh.sh file copied to the server by capistrano after executing the deploy command.
Actually the second line of git-ssh.sh looks like:
exec /usr/bin/ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no "$#"
I can not execute this command directly on the server. The following error occurs:
[5b4fcea9] /tmp/app.de/git-ssh.sh: line 2: /usr/bin/ssh: No such file or directory
After editing the ssh path to /usr/local/bin/ssh it works well but capistrano will upload this file every time calling cap staging deploy.
See my logs on pastie for more details, specially in the git:check part:
http://pastie.org/9523811
It is possible to set this path in my deploy.rb?
Thanks & cheers
Mirko
Yeah, i got it. :))
Rake::Task["deploy:check"].clear_actions
namespace :deploy do
task check: :'git:wrapper' do
on release_roles :all do
execute :mkdir, "-p", "#{fetch(:tmp_dir)}/#{fetch(:application)}/"
upload! StringIO.new("#!/bin/sh -e\nexec /usr/local/bin/ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no \"$#\"\n"), "#{fetch(:tmp_dir)}/#{fetch(:application)}/git-ssh.sh"
execute :chmod, "+x", "#{fetch(:tmp_dir)}/#{fetch(:application)}/git-ssh.sh"
end
end
end
This line seems to be hardcoded in capistrano source code link here.
Instead of changing capistrano source, why don't you create /usr/bin/ssh symlink on the server? So this:
ln -s /usr/local/bin/ssh /usr/bin/ssh
That will create a /usr/bin/ssh symlink that, when executed, will run /usr/local/bin/ssh.
In your Capfile if you add the following line you can override the git tasks:
require 'capistrano/git'
I am build an image using Dockerfile, and I would like to add users to RabbitMQ right after installation. The problem is that during build hostname of the docker container is different from when I run the resultant image. RabbitMQ loses that user; because of changed hostname it uses another DB.
I connot change /etc/hosts and /etc/hostname files from inside a container, and looks that RabbitMQ is not picking my changes to RABBITMQ_NODENAME and HOSTNAME variables.
The only thing that I found working is running this before starting RabbitMQ broker:
echo "NODENAME=rabbit#localhost" >> /etc/rabbitmq/rabbitmq.conf.d/ewos.conf
But then I will have to run docker image with changed hostname all the time.
docker run -h="localhost" image
Any ideas on what can be done? Maybe the solution is to add users to RabbitMQ not on build but on image run?
Just here is example how to configure from Dockerfile properly:
ENV HOSTNAME localhost
RUN /etc/init.d/rabbitmq-server start ; rabbitmqctl add_vhost /test; /etc/init.d/rabbitmq-server stop
This is remember your config.
Yes, I would suggest to add users when the container runs for the first time.
Instead of starting RabbitMQ directly, you can run a wrapper script that will take care of all the setup, and then start RabbitMQ. If the last step of the wrapper script is a process start, remember that you can use exec so that the new process replaces the script itself.
This is how I did it.
Dockerfile
FROM debian:jessie
MAINTAINER Francesco Casula <fra.casula#gmail.com>
VOLUME ["/var/www"]
WORKDIR /var/www
ENV HOSTNAME my-docker
ENV RABBITMQ_NODENAME rabbit#my-docker
COPY scripts /root/scripts
RUN /bin/bash /root/scripts/os-setup.bash && \
/bin/bash /root/scripts/install-rabbitmq.bash
CMD /etc/init.d/rabbitmq-server start && \
/bin/bash
os-setup.bash
#!/bin/bash
echo "127.0.0.1 localhost" > /etc/hosts
echo "127.0.1.1 my-docker" >> /etc/hosts
echo "my-docker" > /etc/hostname
install-rabbitmq.bash
#!/bin/bash
echo "NODENAME=rabbit#my-docker" > /etc/rabbitmq/rabbitmq-env.conf
echo 'deb http://www.rabbitmq.com/debian/ testing main' | tee /etc/apt/sources.list.d/rabbitmq.list
wget -O- https://www.rabbitmq.com/rabbitmq-release-signing-key.asc | apt-key add -
apt-get update
cd ~
wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.5/rabbitmq-server_3.6.5-1_all.deb
dpkg -i rabbitmq-server_3.6.5-1_all.deb
apt-get install -f -y
/etc/init.d/rabbitmq-server start
sleep 3
rabbitmq-plugins enable amqp_client mochiweb rabbitmq_management rabbitmq_management_agent \
rabbitmq_management_visualiser rabbitmq_web_dispatch webmachine
rabbitmqctl delete_user guest
rabbitmqctl add_user bunny password
rabbitmqctl set_user_tags bunny administrator
rabbitmqctl delete_vhost /
rabbitmqctl add_vhost symfony_prod
rabbitmqctl set_permissions -p symfony_prod bunny ".*" ".*" ".*"
rabbitmqctl add_vhost symfony_dev
rabbitmqctl set_permissions -p symfony_dev bunny ".*" ".*" ".*"
rabbitmqctl add_vhost symfony_test
rabbitmqctl set_permissions -p symfony_test bunny ".*" ".*" ".*"
/etc/init.d/rabbitmq-server restart
IS_RABBIT_INSTALLED=`rabbitmqctl status | grep RabbitMQ | grep "3\.6\.5" | wc -l`
if [ "$IS_RABBIT_INSTALLED" = "0" ]; then
exit 1
fi
IS_RABBIT_CONFIGURED=`rabbitmqctl list_users | grep bunny | grep "administrator" | wc -l`
if [ "$IS_RABBIT_CONFIGURED" = "0" ]; then
exit 1
fi
Don't forget to run the container by specifying the right host with the -h flag:
docker run -h my-docker -it --name=my-docker -v $(pwd)/htdocs:/var/www my-docker
The only thing that helped me was to change default value in rabbitmq-env.conf of MNESIA_BASE property to MNESIA_BASE=/data and I added this command RUN mkdir /data in Dockerfile before starting server and add users.
When deploying my rail 3 app with Capistrano It gets to the step where capistrano executes this command:
* executing "if [ -d /var/www/appname/shared/cached-copy ]; then cd /var/www/dflabs1/shared/cached-copy && git fetch -q origin && git fetch --tags -q origi
n && git reset -q --hard d0a1373a3634935de1a75f377698ba53574fe580 && git clean -q -d -x -f; else git clone -q https://github.com/username/dflabs1.git /va
r/www/appname/shared/cached-copy && cd /var/www/appname/shared/cached-copy && git checkout -q -b deploy d0a1373a3634935de1a75f377698ba53574fe580; fi"
servers: ["11.10.1.162"]
Password:
[11.10.1.162] executing command
** [11.10.1.162 :: out] Username for 'https://github.com':
The problem is that when it outputs "Username for 'https://github.com'" , the cursor jumps to a new line without letting me enter the username. If I try and enter the username on the new line the deploy just does nothing. This is happen on an Ubuntu 12.04 desktop to an Ubuntu 12.04 server.
I tried adding the 'set :scm_username' option to deploy.rb but that had no effect. I tried in the Ubuntu terminal and in the Terminal view inside Aptana.
Try to import the SSH-Key of your Deployment-Server into GitHub.
You can do this here: SSH Keys.
You can find the tutorial for SSH-Keys (Github) here: Generating SSH Keys
[EDIT]
Actually I just did it myself and checked the output.
Are you sure you set your set :repository, "git#github.com:[GitHub-User]/[GitHub-Repository-Name].git"in the deploy.rb correctly?
because your
else git clone -q https://github.com/username/dflabs1.git
looks here like
else git clone -q git#github.com:[GitHub-User]/[GitHub-Repository-Name].git