I have a test job which requires gradle to run the test as well as mysql. This example
shows how to use mysql. The problem is the image tag in the link overrides my global image tag of gradle because of which gradle is not found. Is there a way to use multiple images in one job or any other work-around.
This is a shortened version of .gitlab-ci.yml. This is the full one:
image: gradle:jdk11
# Disable the Gradle daemon for Continuous Integration servers as correctness
# is usually a priority over speed in CI environments. Using a fresh
# runtime for each build is more reliable since the runtime is completely
# isolated from any previous builds.
variables:
# Configure mysql service (https://hub.docker.com/_/mysql/)
MYSQL_ROOT_PASSWORD: mysql
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
## build configs....
test_MariaDBImpl:
needs:
- build_MariaDBImpl
stage: test
services:
- mysql
image: mysql
script:
- echo "create user if not exists 'test'#'localhost'; grant all privileges on *.* to 'test'#'localhost'; flush privileges;" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql
- gradle cleanTest :MariaDBImpl:test
Edit 1: One work around that I thought of was to add a before_script and download either gradle or mysql in it but I guess there is a better way?
Edit 2: Or maybe create a docker image with all three and use that one instead?
That's the expected behaviour with GitLab-CI. You set image: gradle:jdk11 as default image to use for the CI job unless overridden in the job configuration like you did with image: mysql.
As you said you have several options here if you need both gradle and mysql-client.
Build your own Docker image that contains both
Remove image: mysql, use the default image: gradle:jdk11 and install mysql-client at the beginning of your script block or with before_script. gradle:jdk11 is based on Ubuntu so you can run: apt-get update && apt-get install -y mysql-client
Keep image: mysql and install gradle with apt-get update && apt-get install -y gradle (I don't recommend that, installing gradle via APT install a lot of packages and will slow your CI job down unnecessarily)
Keep image: mysql, add gradle wrapper to your repository and directly call it from gitlab-ci. Edit: This is not very useful as it would also require to download the JDK to build the project. It works best if the base image already contains it.
My preference would be to keep on using gradle:jdk11 as base image and just install the mysql-client package for that specific job since your other jobs only need gradle.
I am using spark-shell and I am unable to pick up external jars. I run spark in EMR.
I run the following command:
spark-shell --jars s3://play/emr/release/1.0/code.jar
I get the following error:
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Warning: Skip remote jar s3://play/emr/release/1.0/code.jar
Thanks in advance.
This is a limitation of Apache Spark itself, not specifically Spark on EMR. When running Spark in client deploy mode (all interactive shells like spark-shell or pyspark, or spark-submit without --deploy-mode cluster or --master yarn-cluster), only local jar paths are allowed.
The reason for this is that in order for Spark to download this remote jar, it must already be running Java code, at which point it is too late to add the jar to its own classpath.
The workaround is to download the jar locally (using the AWS S3 CLI) then specify the local path when running spark-shell or spark-submit.
You can do this with a spark-shell command line on the EMR box itself:
spark-submit --verbose --deploy-mode cluster --class com.your.package.and.Class s3://bucket/path/to/thejar.jar 10
You can also call this command using the AWS Java EMR Client Library or the AWS CLI. The key is to use: '--deploy-mode cluster'
Had same issue, you can add "--master yarn --deploy-mode cluster" args and it will allows you to execute s3 jars remotely
I am trying to run the Tooltwist Controller from the command line using:
./ttc/config/bin/checkLaunchpad.sh controllerTest
But got below error:
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in gem_original_require': no such file to load -- xmlsimple (LoadError)
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:inrequire'
from /tooltwist/tooltwist_osx_7.0/devel/ttc/config/bin/checkLaunchpad.rb:21
The ToolTwist controller uses a Ruby script to determine which version of payloads should be used, and to synchronize the git caches. This script uses a Ruby gem (like a library) to parse the config files' XML. To install this gem, simply type this on the command line:
gem install xml-simple
i am following the guide at http://guides.spreecommerce.com/getting_started.html
all's ok until i run
bundle exec rake spree_sample:load
rake says
Don't know how to build task 'spree_sample:load'
if i decide to skip that and do
rake db:bootstrap
it says
Table 'myproject_development.countries' doesn't exist
if alternatively i run
rake db:migrate
rake db:seed
rake spree_sample:load
it also says
Don't know how to build task 'spree_sample:load'
if i still continue that and run
rake db:admin:create
i get another error:
no such file to load -- /home/pavel/cetsy/db/sample/users.rb
bundler information:
...
Using rails (3.0.9)
...
Using spree_core (0.60.1)
Using spree_auth (0.60.1)
Using spree_api (0.60.1)
Using spree_dash (0.60.1)
Using spree_promo (0.60.1)
Using spree_sample (0.60.1)
Using spree (0.60.1)
Simon was right, upgrading rails and spree to 3.1.1/0.70 partially solved the issue
now, running
bundle exec rake spree_sample:load
still produces
Don't know how to build task 'spree_sample:load'
but skipping this step lets me finish installing spree without other problems
I'm working on a Ruby on Rails 3 webapp on Heroku. How do I empty the database?
To drop the database, if you are using SHARED_DATABASE_URL:
$ heroku pg:reset DATABASE_URL
Now to recreate the database with nothing in it:
$ heroku run rake db:migrate
To populate the database with your seed data:
$ heroku run rake db:seed
---OR---
You can combine the last two (migrate & seed) into one action by executing this:
$ heroku run rake db:setup
Edit 2014-04-18: rake db:setup doesn't work with Rails 4, it fails with a Couldn't create database error.
Edit 2014-10-09: You can use rake db:setup with Rails 4. It does give you a Couldn't create database error (because the database was already created using the heroku pg:reset command). But it also loads your database schema and your seeds after the error message.
You can do this with pretty much any rake command, but there are exceptions. For example, db:reset doesn't work via heroku run rake. You have to use pg:reset instead.
More information can be found in Heroku's documentation:
Running Rake Commands
Reset Postgres DB
Heroku has deprecated the --db option now, so now use:
heroku pg:reset DATABASE_URL --confirm {the name of your app}
It's a little confusing because you use the literal text SHARED_DATABASE but where I have written {the name of your app} substitute the name of your app. For example, if your app is called my_great_app then you use:
heroku pg:reset DATABASE_URL --confirm my_great_app
To drop the database:
$ heroku pg:reset SHARED_DATABASE --confirm NAME_OF_THE_APP
To recreate the database:
$ heroku run rake db:migrate
To seed the database:
$ heroku run rake db:seed
**Final step
$ heroku restart
The current, ie. 2017 way to do this is:
heroku pg:reset DATABASE
https://devcenter.heroku.com/articles/heroku-postgresql#pg-reset
Now the command is
heroku pg:reset DATABASE_URL --confirm your_app_name
this way you can specify which app's db you want to reset.
Then you can run
heroku run rake db:migrate
heroku run rake db:seed
or direct for both above commands
heroku run rake db:setup
And now final step to restart your app
heroku restart
I contacted Heroku support, and they confirmed that it is a bug with the latest gem (I am using heroku-2.26.2)
Charlie - we are aware of this issue with the 'heroku' gem and are
working to fix it.
Here's the issue if you care to follow-along -
https://github.com/heroku/heroku/issues/356
Downgrading to an earlier version of the 'heroku' gem should help. I've been using v2.25.0 for most of today without issue.
Downgrade with the following commands:
gem uninstall heroku
gem install heroku --version 2.25.0
If you already have multiple gems installed, you may be presented with:
Select gem to uninstall:
1. heroku-2.25.0
2. heroku-2.26.2
3. All versions
Just uninstall #2 and rerun the command. Joy!
The complete answer is (for users with multi-db):
heroku pg:info - which outputs
=== HEROKU_POSTGRESQL_RED <-- this is DB
Plan Basic
Status available
heroku pg:reset HEROKU_POSTGRESQL_RED --confirm app_name
More information found in: https://devcenter.heroku.com/articles/heroku-postgresql
Now it's diffrent with heroku. Try:
heroku pg:reset DATABASE --confirm
Now it's also possible to reset the database through their web interface.
Go to dashboard.heroku.com select your app and then you'll find the database under the add-ons category, click on it and then you can reset the database.
Today the command
heroku pg:reset --db SHARED_DATABASE_URL
not working for shared plans, I'm resolve using
heroku pg:reset SHARED_DATABASE
Check your heroku version. I just updated mine to 2.29.0, as follows:
heroku --version
#=> heroku-gem/2.29.0 (x86_64-linux) ruby/1.9.3
Now you can run:
heroku pg:reset DATABASE --confirm YOUR_APP_NAME
Then create your database and seed it in a single command:
heroku run rake db:setup
Now restart and try your app:
heroku restart
heroku open
Login to your DB using
heroku pg:psql and type the following commands:
drop schema public cascade;
create schema public;
In case you prefer to use Heroku Web-site:
Go to https://postgres.heroku.com/databases
Select the database you want to reset
Click on a settings button in the right upper corner
Click "Reset Database" as shown below:
type in "RESET" and press ok
This is what worked for me.
1.clear db.
heroku pg:reset --app YOUR_APP
After running that you will have to type in your app name again to confirm.
2.migrate db to recreate.
heroku run rake db:migrate --app YOUR_APP
3.add seed data to db.
heroku run rake db:seed --app YOUR_APP
Assuming you want to reset your PostgreSQL database and set it back up, use:
heroku apps
to list your applications on Heroku. Find the name of your current application (application_name). Then run
heroku config | grep POSTGRESQL
to get the name of your databases. An example could be
HEROKU_POSTGRESQL_WHITE_URL
Finally, given application_name and database_url, you should run
heroku pg:reset `database_url` --confirm `application_name`
heroku run rake db:migrate
heroku restart
If you are logged in from the console, this will do the job in the latest heroku toolbelt,
heroku pg:reset --confirm database-name
I always do this with the one-liner 'heroku pg:reset DATABASE'.
Best solution for you issue will be
heroku pg:reset -r heroku --confirm your_heroku_app_name
--confirm your_heroku_app_name
is not required, but terminal always ask me do that command.
After that command you will be have pure db, without structure and stuff, after that you can run
heroku run rake db:schema:load -r heroku
or
heroku run rake db:migrate -r heroku