I'm trying to provision an EMR with a bootstrap action. I can see the stdout log and it finishes fine.
The last action is install boto3.
Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.18.28 botocore-1.21.28 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.0
However after that EMR fails with "On the master instance, application provisioning failed". See log below.
I think this might be due to what I install in the bootstrap. java 11, python 3.7 etc. However, If run the same script manually via SSH after EMR has been provisioned everything works fine. Is there any way to execute the bootstrap action after all applications have been installed?
Error log: from provision-node/apps-phase/0/60c849d6-ca64-486d-8b4a-4c60201b168f/
2021-08-25 15:01:07,025 ERROR main: Encountered a problem while provisioning
com.amazonaws.emr.node.provisioner.puppet.api.PuppetException: Unable to complete transaction and some changes were applied.
at com.amazonaws.emr.node.provisioner.puppet.api.ApplyCommand.handleExitcode(ApplyCommand.java:74)
at com.amazonaws.emr.node.provisioner.puppet.api.ApplyCommand.call(ApplyCommand.java:56)
at com.amazonaws.emr.node.provisioner.bigtop.BigtopPuppeteer.applyPuppet(BigtopPuppeteer.java:73)
at com.amazonaws.emr.node.provisioner.bigtop.BigtopDeployer.deploy(BigtopDeployer.java:22)
at com.amazonaws.emr.node.provisioner.NodeProvisioner.provision(NodeProvisioner.java:25)
at com.amazonaws.emr.node.provisioner.workflow.NodeProvisionerWorkflow.doWork(NodeProvisionerWorkflow.java:196)
at com.amazonaws.emr.node.provisioner.workflow.NodeProvisionerWorkflow.work(NodeProvisionerWorkflow.java:101)
at com.amazonaws.emr.node.provisioner.Program.main(Program.java:30)
There is a way to run post provisioning (second stage) bootstrapping actions on an EMR. It's a bit of a hack and it works like this.
As your last bootstrapping action you need to copy the file you want to run after the installation of stuff like Hadoop or Spark from S3 into the node and run it as a background action.
The background action will wait for the node to be fully provisioned before it actually runs the code you wanted to run in the first place before it exits the loop.
Here's the code:
set_up_post_provisioning.sh
#!/bin/bash -x
aws s3 cp s3://path/to/bootstrap/scripts/post_provisioning.sh /home/hadoop/post_provisioning.sh &&
sudo bash /home/hadoop/post_provisioning.sh &
exit 0
post_provisioning.sh
#!/bin/bash
while true
do
NODEPROVISIONSTATE=$(sed -n '/localInstance [{]/,/[}]/{
/nodeProvisionCheckinRecord [{]/,/[}]/ {
/status: / { p }
/[}]/a
}
/[}]/a
}' /emr/instance-controller/lib/info/job-flow-state.txt | awk '{ print $2 }')
if [[ "$NODEPROVISIONSTATE" == "SUCCESSFUL" ]]
then
sleep 10
echo "Your code here"
exit
fi
sleep 10
done
Make sure that only set_up_post_provisioning.sh is an actual bootstrap action, since the next stage will not start if all the bootstrap actions are not finished.
I hope it helps!
Related
I'm using Laravel Forge to run a simple deploy script. npm run build calls 'vue-cli-service build'.
Script below. The script 'ends' on
DONE Build complete. The dist directory is ready to be deployed.
INFO Check out deployment instructions at https://cli.vuejs.org/guide/deployment.html
but the thread does not quit, which causes issues in forge (e.g. thinks it's timed out or failed when it hasn't).
How do I do the equivelant of ctrl-c in a terminal once this has finished, in the deploy script? I've seen threads on trap SIGINT / trap etc. but I'm still not really sure how to implement it.
It may be that I just include the exit callback fix noted here: Vue-cli-service serve build completion callback?
git pull origin $FORGE_SITE_BRANCH;
npm run build;
( flock -w 10 9 || exit 1
echo 'Restarting FPM...'; sudo -S service $FORGE_PHP_FPM reload ) 9>/tmp/fpmlock
if [ -f artisan ]; then
$FORGE_PHP artisan migrate --force
fi```
Try out to add Daemon termination command to the end of your deployment script
$FORGE_PHP artisan horizon:terminate
We have a large number of runners running a large number of jobs in one of our Gitlab CI/CD pipelines.
Each of these runners has a concurrency of 1, and they are of executor type shell.
[EDIT] These runners are AWS EC2 instances using Amazon Linux 2.
After certain jobs in the pipeline have completed, I would like them to run a reboot command to restart the runner.
However, some of these jobs will be tests. Currently, when I run the reboot command, the job fails. Obviously I can allow_failure so that the job passes, but this then means we have no way of determining whether or not the actual test has passed.
Originally, my test job looked like this:
after_script:
- sleep 1 && reboot
I have also tried the following variations:
after_script:
- sleep 15 && reboot
- exit 0
after_script:
- (sleep 15 ; reboot ) &
- exit 0
I've also tried running a shell script with the same contents.
All of these result in the same problem - ERROR: Job failed (system failure): aborted: terminated.
Can anyone think of a clever way round this?
In the end, I had to run this in a screen:
sudo screen -dm bash -c 'sleep 5; shutdown -r now;'
This allowed me, in a Gitlab CI pipeline, to run this as a script element, and immediately afterwards execute an exit command, like this:
after_script:
- sudo screen -dm bash -c 'sleep 5; shutdown -r now;'
- exit 0
This way, if a test fails - the job fails. If a test passes, the job passes. No need for allow_failure.
Unfortunately... I'm unsure of how to then contend with artifacts which take place after the after_script commands. If anyone has any ideas about that one, please add a comment here.
updated: added the missing docker attach.
Hi am trying to run a docker container, with -dti. but I cannot access with a terminal set to dumb. is there a way to change this (it is currently set to xterm, even though my ssh client is dumb)
example:
create the container
docker run -dti --name test -v /my-folder alpine /bin/ash
docker attach test
apk --update add nodejs
cd /my-folder
npm install -g gulp
the last command always contains ascii escape chars to move the cursor.
I have tried "export TERM=dumb" inside the running container, but it does not work.
is there a way to "run" this using the dumb terminal?
I am running this from a script on another computer, via (dumb) ssh.
using the -t which sets this https://docs.docker.com/engine/reference/run/#env-environment-variables, however removing effects the command prompt (the prompt is not shown)
possible solution 1 remove the -t and keep the -i. To see if the command has completed echo out a known token (ENDENDEND). ie
docker run -di --name test -v /my-folder alpine /bin/ash
docker attach test
apk --update add nodejs;echo ENDENDEND
cd /my-folder;echo ENDENDEND
npm install -g gulp;echo ENDENDEND
not pretty, but it works (there is no ascii in the results)
Possible solution 2 use the journal, docker can log out to the linux journal, this can be gathered as commands are executed in the container. (I have yet to fully test this one out. however the log seems to be a nicer output of what happened)
update:
Yep -t is the problem.
However if you want to see the entire process when running a command, maybe this way is better:
docker run -di --name test -v/my-folder alpine /bin/ash
docker exec -it test /bin/ash
finally you need to kill the container after all jobs finished.
docker run -d means "Run container in background and print container ID"
not start the container as a daemon
I was hitting this issue on OSx running docker, i had to do 2 things to stop the terminal/ascii/ansi escape sequences.
remove the "t" option on the docker run command (from docker run -it ... to docker run -i...)
ensure to force bash or sh shells used on osx when running the command from a script file, not the default zsh
Also
the escape sequences were not always visible on the terminal
even so, they still usually caused content corruption, even with SED brought to bear
they always were shown in my editor
When creating a PKGBUILD file one can execute hooks at pre_install(), post_install(), etc.
I now have a custom arch linux pacman package that I need some custom checks done before it is installed to determine if it is safe to install or not.
I would like to run my test in the pre_istall() script and have pacman abort the installation if I say so in the script.
So, how can this be accomplished? So far all I have accomplished is getting an error message in the log but pacman continues with the istall...
I would not recommend this as it sounds like a code smell: in my opinion the pre_install() hook is designed to perform actions before package files are actually installed on your drive, but it is not meant to check whether the package should be installed.
In my opinion, such a check belongs to some other place out of the package.
You could call a command, which returns a non-zero exit-code, to cancel the build process. The simplest command I could think of is sh -c "exit 1", since just exit 1 results in an immediate exit without any proper cleanup.
Here is a simple example that checks if a file exists and cancels the build process if not:
prepare() {
if ! [ -f "/usr/bin/ffmpeg" ]; then
echo "Error: FFmpeg executable '/usr/bin/ffmpeg' is missing."
sh -c "exit 1"
fi
}
However, galaux is right. Usually such checks should happen upstream.
I'm trying to make a push notification work on my debian vps (apace2, mysql).
I use a php script from this tutorial (http://www.raywenderlich.com/3525/apple-push-notification-services-tutorial-part-2).
Basically, the script is put in an infintive loop, that check a mysql table for new records every couple of seconds. The tutorial says it should be run as a background process.
// This script should be run as a background process on the server. It checks
// every few seconds for new messages in the database table push_queue and
// sends them to the Apple Push Notification Service.
//
// Usage: php push.php development &
So I have four questions.
How do I start the script from the terminal? What should I type? The script location on the server is:
/var/www/development_folder/scripts/push2/push.php
How can I kill it if I need to (without having to restart apace)?
Since the push notification is essential, I need a way to check if the script is running.
The code (from the tutorial) calls a function is something goes wrong:
function fatalError($message)
{
writeToLog('Exiting with fatal error: ' . $message);
exit;
}
Maybe I can put something in there to restart the script? But It would also be nice to have a cron job or something that check every 5 minute or so if the script is running, and start it if it doens't.
4 - Can I make the script automatically start after a apace or mysql restart? If the server crash or something else happens that need a apace restart?
Thanks a lot in advance
You could run the script with the following command:
nohup php /var/www/development_folder/scripts/push2/push.php > /dev/null &
The nohup means that that the command should not quit (it ignores hangup signal) when you e.g. close your terminal window. If you don't care about this you could just start the process with "php /var/www/development_folder/scripts/push2/push.php &" instead. PS! nohup logs the script output to a file called nohup.out as default, if you do not want this, just add > /dev/null as I've done here. The & at the end means that the proccess will run in the background.
I would only recommend starting the push script like this while you test your code. The script should be run as a daemon at system-startup instead (see 4.) if it's important that it runs all the time.
Just type
ps ax | grep push.php
and you will get the processid (pid). It will look something like this:
4530 pts/3 S 0:00 php /var/www/development_folder/scripts/push2/push.php
The pid is the first number you'll see. You can then run the following command to kill the script:
kill -9 4530
If you run ps ax | grep push.php again the process should now be gone.
I would recommend that you make a cronjob that checks if the php-script is running, and if not, starts it. You could do this with ps ax and grep checks inside your shell script. Something like this should do it:
if ! ps ax | grep -v grep | grep 'push.php' > /dev/null
then
nohup php /var/www/development_folder/scripts/push2/push.php > /dev/null &
else
echo "push-script is already running"
fi
If you want the script to start up after booting up the system you could make a file in /etc/init.d (e.g. /etc.init.d/mypushscript with something like this inside:
php /var/www/development_folder/scripts/push2/push.php
(You should probably have alot more in this file)
You would also need to run the following commands:
chmod +x /etc/init.d/mypushscript
update-rc.d mypushscript defaults
to make the script start at boot-time. I have not tested this so please do more research before making your own init script!